Project

General

Profile

Statistics
| Revision:

# Date Author Comment
9095 02/06/2015 09:15 AM ben leinfelder

use HttpMultipartRestClient since the DefaultHttpMultipartRestClient was removed from d1_libclient_java

8959 11/10/2014 02:22 PM Jing Tao

During the replication, the remote content will be saved without alteration.

8810 07/23/2014 04:19 PM ben leinfelder

add support for v2 DataONE API.

8790 05/20/2014 04:08 PM ben leinfelder

show the SM and ORE generation buttons even if they have not registered/configured dataone. many potential MNs want to see their generated SM before registering (and we want them to too!).

8760 04/30/2014 01:02 PM Lauren Walker

Add admin service to update DOI registrations by specifying a list of formatIds or DOIs, or update all.

8746 04/23/2014 12:35 PM ben leinfelder

first pass at allowing admins to update DOI registration. This only acts on EML objects at the moment and is meant to illustrate one mechanism for updating the DOIs. https://projects.ecoinformatics.org/ecoinfo/issues/6530

8739 04/17/2014 02:19 PM Jing Tao

Index the document after it has been inserted.

8738 04/17/2014 02:18 PM Jing Tao

Index the document after document is written to the db.

8689 03/03/2014 03:41 PM ben leinfelder

support content from all serverLocations when summarizing entity info (semtools)

8647 02/25/2014 04:14 PM ben leinfelder

recursively submit obsoleted objects for indexing when instructed. https://projects.ecoinformatics.org/ecoinfo/issues/6424

8590 02/11/2014 01:29 PM Peter Slaughter

Run syncAll in a single thread so admin config UI doesn't freeze

8585 02/10/2014 11:52 AM ben leinfelder

Couple modifications:
-use "pid" throughout so as not to confuse docids and pids
-ensure any failures in the set do not prevent synching for other pids in the set

8491 01/12/2014 04:48 PM Peter Slaughter

Sync access policy between mn -> cn in case where metacat native ui being used to update ap on mn

8464 01/07/2014 01:56 PM ben leinfelder

Unify solr indexing with an IndexTask that is added to the queue -- allows us to send more than just the systemMetadata to the indexer. Initially this is for READ event counts for each document. https://projects.ecoinformatics.org/ecoinfo/issues/6346

8304 10/09/2013 11:52 PM Matt Jones

Reviewed code for all uses of FileInputStream, checking to see if the method should be closing the stream, and if so, closing it in the method as well as in the finally clause to ensure we don't leak file descriptors.

8301 10/09/2013 02:42 PM Matt Jones

Closing some more streams that were left open. This Bug #6136 seems to be pervasive and is going to require an extensive audit to find all of the places where streams are not closed properly.

8299 10/09/2013 01:47 PM Matt Jones

Refactor to use IOUtils.closeQuietly() which handles nulls and streams that are already closed.

8297 10/09/2013 12:44 PM Matt Jones

Closing FileOutputStream handles so that the OS limits on filehandles are not exceeded.

7840 07/02/2013 04:47 PM ben leinfelder

support a "force replication delete all action" during replication. This is used when we want Metacat to remove the content from the other target replicas because the DataONE delete() action was called (more powerful than just "archive").

7812 06/20/2013 04:49 PM ben leinfelder

use an independent ISet<SystemMetadata> structure to communicate objects that should be indexed by metacat-index. https://projects.ecoinformatics.org/ecoinfo/issues/5943

7638 04/30/2013 04:47 PM ben leinfelder

add space to prevent syntax error when additional clause is appended. https://projects.ecoinformatics.org/ecoinfo/issues/5929.

7636 04/30/2013 02:50 PM Matt Jones

CHange replication 'update' query to use a LEFT JOIN so that the performance of the replication update action is improved, which had been causing an HTTP timeout for large metacat installations. See https://projects.ecoinformatics.org/ecoinfo/issues/5929.

7519 03/22/2013 12:29 PM ben leinfelder

include xml_revisions.
do not allow removal of server_location = 1 documents (these are not replicas).
https://redmine.dataone.org/issues/3539

7475 12/12/2012 02:38 PM ben leinfelder

move DocInfo parsing into utilities project so that it can be used by Morpho as well as Metacat.
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5737

7382 09/14/2012 02:01 PM ben leinfelder

use correct docid format when checking for existing mappings.

7379 09/12/2012 02:22 PM ben leinfelder

use CDATA for docname field in docInfo so that XML parser ignores the content that can contain characters like "&

7370 09/04/2012 03:43 PM ben leinfelder

use SchemaLocationResolver to fetch remote entries for the xml_catalog -- we want to be able to fetch included xsd files as well as use any error handling it provides for checking the schemas.

7358 08/23/2012 09:45 PM ben leinfelder

create docid-guid mapping during replication if it does not exist. we were [incorrectly] assuming that there would be SM coming with the document info that would fill this information in, but for traditional non-MN Metacat deployments there is no SM to provide a mapping. In this case we use the docid as the guid.

7356 08/17/2012 12:42 PM ben leinfelder

stream the replication "update" response rather than building up a complete list in a stringbuffer. prompted by findings on t he CN: https://redmine.dataone.org/issues/3141

7348 08/06/2012 11:08 AM ben leinfelder

remove unused "dataonelogger"

7338 08/01/2012 07:00 PM ben leinfelder

remove possibility for infinite loop in case data replication is not configured for the server and a data file is encountered (yikes!)

7337 08/01/2012 05:33 PM ben leinfelder

added logging debug statements to see where the replication timeout might be occurring.

7317 07/17/2012 12:06 PM ben leinfelder

only look up the client timeout property once, not every time we make a call
https://redmine.dataone.org/issues/3078

7313 07/17/2012 10:19 AM ben leinfelder

configurable replication client timeout
https://redmine.dataone.org/issues/3078

7196 05/29/2012 09:33 AM ben leinfelder

stack trace the HZ put exception during CN-CN replication

7195 05/29/2012 07:37 AM ben leinfelder

additional debugging statements for CONCURRENT_MAP_PUT error during CN-CN replication.

7175 05/18/2012 12:09 PM ben leinfelder

instead of generating SM and ORE maps during dataone configuration/MN registration, moved this all to the replication admin screen where we can target generation for specific nodes. That way it's more controlled as to when and where we generate DataONE required content....

7170 05/14/2012 02:36 PM ben leinfelder

add "Generate System Metadata" button to the replication server list display. When clicked, we generate SM for records belonging to that source server. This is only enabled when DataONE has been configured.
https://redmine.dataone.org/issues/2762

7150 04/30/2012 04:03 PM ben leinfelder

optionally remove the document/data file from the filesystem completely when 'deleting' it.
https://redmine.dataone.org/issues/2677

7128 04/09/2012 03:18 PM ben leinfelder

add a parameter for optionally writing EML-embedded access control rules to the Metacat DB.
https://redmine.dataone.org/issues/2584
https://redmine.dataone.org/issues/2583

7120 04/05/2012 04:28 PM ben leinfelder

band-aid for CN-CN replication permOrder issue when access control is embedded in EML and the system metadata is replicated before the EML. we just log the inconsistency and allow the insert to succeed
https://redmine.dataone.org/issues/2583

7045 03/07/2012 02:30 PM ben leinfelder

check whether mapping (

6968 02/01/2012 09:44 AM ben leinfelder

process systemMetadata from the docInfo string before writing to the database so that we guarantee guid-docid mapping exists before attempting to look it up.

6814 12/21/2011 01:00 PM ben leinfelder

remove flag for independent system metadata replication -- these entries are replicated along with the data/metadata objects or via hazelcast when the actual object is not on the server.

6783 12/13/2011 05:06 PM ben leinfelder

only create guid->docid mapping during metadata replication if it does not already exist
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5520

6782 12/13/2011 04:50 PM ben leinfelder

do not treat access change as an update -- it should not attempt to retrieve the contents of the object
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5520

6780 12/13/2011 04:18 PM ben leinfelder

only create guid->docid mapping during data replication if it does not already exist
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5520

6775 12/13/2011 11:34 AM ben leinfelder

process system metadata before access rules (access control is now driven by GUID so the mapping needs to be there)

6708 11/29/2011 01:05 PM ben leinfelder

use shared method for looking up "docInfo" map -- both in Metacat replication and in D1 system metadata generation

6672 11/17/2011 02:47 PM ben leinfelder

replication control panel now fully implemented as an admin configuration screen
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5528

6669 11/17/2011 12:19 PM ben leinfelder

move replication configuration actions to the admin servlet and out of the replication servlet
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5528

6666 11/17/2011 11:19 AM ben leinfelder

save SystemMetadata when replicating data and metadata -- this way if/when the node decides to be a DataONE MN it already has the information needed for each object

6628 11/09/2011 04:26 PM ben leinfelder

get server param only when it is expected

6627 11/09/2011 03:34 PM ben leinfelder

check replication table (not keystore) for trusted server host name match

6620 11/08/2011 11:31 AM ben leinfelder

started replication unit test

6616 11/07/2011 03:53 PM ben leinfelder

add note about alternative methods for getting cert/key

6614 11/07/2011 02:31 PM ben leinfelder

use DateTimeMarshaller for all replication date transfers

6613 11/07/2011 12:31 PM ben leinfelder

print the stacktrace when there is an error -- debuggin!

6611 11/07/2011 10:55 AM ben leinfelder

use SSL to get content from stream

6595 11/02/2011 08:40 PM ben leinfelder

http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5527

6587 11/01/2011 07:35 PM ben leinfelder

skip verification -- remaining TODO

6586 11/01/2011 07:08 PM ben leinfelder

verify certificate

6585 11/01/2011 06:41 PM ben leinfelder

inspect keystore entries for matching client certificate

6584 11/01/2011 05:27 PM ben leinfelder

lookup the correct property for keystore file

6582 11/01/2011 03:23 PM ben leinfelder

use HttpClient to set up SSL connection when doing replication calls -- this will use the server's configured certificate as the client certificate on the request. The server it is calling can then inspect that certificate and decide whether or not it trusts the caller.

6579 11/01/2011 01:33 PM ben leinfelder

check client-provided certificate when servicing ReplicationServlet requests.

6542 10/20/2011 02:03 PM ben leinfelder

add User-Agent logging to support D1 requirements

6531 10/12/2011 11:08 AM ben leinfelder

include SystemMetadata when replicating data and metadata documents -- this allows us to establish the guid-to-docid mapping that is crucial for being able to read the replicated document by guid (d1 api)

6448 09/16/2011 03:17 PM ben leinfelder

do not send <systemMetadata> with the <docInfo> replication information - this is handled by the Hazelcast shared map

6447 09/16/2011 03:07 PM ben leinfelder

rely on Hazelcast to store the SystemMetadata locally for the node. Entry event listeners store the shared system metadata on their local node when alerted. TODO: remove old replication code that included system metadata xml when replicating scimeta and data

6367 07/28/2011 10:10 AM ben leinfelder

remove ServiceTypeUtil - replace with TypeMarshaller

6366 07/27/2011 04:25 PM ben leinfelder

use new "v1" types from DataONE

6335 07/11/2011 12:19 PM ben leinfelder

use correct log name for the class

6135 06/08/2011 05:08 PM ben leinfelder

add option for replicating system metadata (dataone)
https://redmine.dataone.org/issues/1626

6125 06/07/2011 10:27 AM ben leinfelder

force replication for newly-registered system metadata

6124 06/07/2011 09:53 AM Chris Jones

Merged in the D1_0_6_2_BRANCH changes that include the transition from ObjectFormat calls to ObjectFormatCache calls.

6119 06/02/2011 04:40 PM ben leinfelder

include System Metadata forced replication - just need to figure out when to call it!

6118 06/01/2011 01:45 PM ben leinfelder

handle timed replication of system metadata. there are still a few outstanding issues:
-track server location of system metadata-only entries
-replication policy flag for system metadata-only entries?
-locking for replicated entries?
-forced replication of entries

6102 05/25/2011 03:53 PM ben leinfelder

transfer full System Metadata (as XML) during document and data replication

6099 05/25/2011 11:59 AM ben leinfelder

-remove system metadata guid -> local id mapping (there is no document for system metadata now)
-include system metadata elements when replicating data objects (TODO: transfer all system metadata structures with the docinfo request).
TODO: remove docid+rev from the systemMetadata table definition

6097 05/24/2011 04:18 PM ben leinfelder

do not use XML files for storing SystemMetadata - use DB tables only.

6079 05/05/2011 03:14 PM ben leinfelder

use update method to update the mapping between local and guid (d1) when we get a force replication request that is an "update

6057 05/03/2011 04:04 PM ben leinfelder

use "object_format" element consistently so that it is replicated across instances
https://redmine.dataone.org/issues/1514

6015 03/22/2011 05:19 PM ben leinfelder

insert/update documents with null user and null group to circumvent access control restrictions then update the user_owner and user_updated values to reflect what exists on the originating server (pisco)

6014 03/22/2011 01:24 PM ben leinfelder

use 'user_updated' field when writing the replicated document - allows most recent ownership/permissions to be used (in case LDAP groups have shifted) and is more accurate for both updates and initial inserts (hopefully addresses the replication issue we are having with pisco)

6001 03/02/2011 02:12 PM Chris Jones

DocumentImpl.delete() now throws finer grained exceptions (not a general exception). Consequently, the classes that call it have been updated to handle the thrown exceptions, including CrudService, ReplicationHandler, and ReplicationService.

5950 02/11/2011 01:48 PM berkley

adding more debuggin and fixing bug with systemmetadata

5944 02/10/2011 04:17 PM berkley

fixed replication bug where systemmetadata was not getting procssed correctly

5930 02/08/2011 12:05 PM berkley

fixed typo that prevented replication

5917 02/04/2011 12:39 PM berkley

fixed bugs in listObjects

5895 02/03/2011 03:25 PM berkley

added code to do database query for listObjects

5755 12/23/2010 09:10 AM ben leinfelder

only call response.getWriter() when we are about to send text/xml to the client, otherwise we end of calling both getWriter() and getOutputStream() - resulting in an illegal state.

5752 12/21/2010 02:26 PM ben leinfelder

use detected XML encoding when reading/writing files
use UTF-8 as default when performing queries in the DB (assume DB is using UTF-8)
remove as many PrintWriters (uses system default character encoding only) as possible and construct OutputStreamWriters where explicit encoding can be given....

5709 12/08/2010 04:59 PM ben leinfelder

add support for EML 2.1.1

5459 07/28/2010 10:00 AM berkley

reformatting logs for robert

5458 07/27/2010 11:25 AM berkley

added another logging statement

5457 07/27/2010 11:04 AM berkley

added replicate log statements with the guid and localId

5455 07/26/2010 08:21 PM berkley

hopefully fixed bug with systemmetadata replication

5454 07/26/2010 08:04 PM berkley

debug statements in dbsaxhandler

5451 07/26/2010 02:05 PM berkley

fixing problems with replication and systemmetadata