Removed a unused parameter - formatId from method writeReplication.
Disable the feature of downloading external schemas.
Replaced the JiBXException by MarshallingException.
If the client certificate is null, Metacat will throw an exception.
Add more information on the error message if the client's certificate is not trusted.
Made the ssl http client not moinitor the stale connections.
Use the new AutoCloseInputStream wrapper for replication. It will close but http client and input stream when the inputStream.close method is called.
If a format id in the system metadata is registered in the xml_catalog table, we will use the schema location for the format id to validate the xml instance;otherwise, we will use our previous way.
Add the code to check if the metacat is in the read-only mode.
Don't use ssClient as a class field. Now it is a local variable. This fixed an issue that two statements using the client caused the exception - java.net.SocketTimeoutException: Read timed out.
refs: #7092. found another unclosed inputStream, this time in ReplicationService.handleForceReplicationDataFileRequest(...).
fixes #7092: added finally clause to ReplicationHandler.update method to unconditionally close the inputStreams it keeps in a Vector. Needs testing.
ReplicationServices.getUrlStream(url) callers were not closing the inputstream properly, as was revealed when switching to libclient v2 (a pooling connection manager in HttpClient) and encountering reasource leaks.
bullet-proofed exception handling in ReplicationService.getURLStream, to make sure client communication exceptions are handled and logged.
fixed getSSLClient method.
fixed class name typo.
fixed private getSSLClient method for HttpClient v4.3 compatibility. Now also caching the RestClient for reuse to save overhead.
organize imports. move exception-throwing method call into try block.
use HttpMultipartRestClient since the DefaultHttpMultipartRestClient was removed from d1_libclient_java
During the replication, the remote content will be saved without alteration.
add support for v2 DataONE API.
show the SM and ORE generation buttons even if they have not registered/configured dataone. many potential MNs want to see their generated SM before registering (and we want them to too!).
Add admin service to update DOI registrations by specifying a list of formatIds or DOIs, or update all.
first pass at allowing admins to update DOI registration. This only acts on EML objects at the moment and is meant to illustrate one mechanism for updating the DOIs. https://projects.ecoinformatics.org/ecoinfo/issues/6530
Index the document after it has been inserted.
Index the document after document is written to the db.
support content from all serverLocations when summarizing entity info (semtools)
recursively submit obsoleted objects for indexing when instructed. https://projects.ecoinformatics.org/ecoinfo/issues/6424
Run syncAll in a single thread so admin config UI doesn't freeze
Couple modifications:-use "pid" throughout so as not to confuse docids and pids-ensure any failures in the set do not prevent synching for other pids in the set
Sync access policy between mn -> cn in case where metacat native ui being used to update ap on mn
Unify solr indexing with an IndexTask that is added to the queue -- allows us to send more than just the systemMetadata to the indexer. Initially this is for READ event counts for each document. https://projects.ecoinformatics.org/ecoinfo/issues/6346
Reviewed code for all uses of FileInputStream, checking to see if the method should be closing the stream, and if so, closing it in the method as well as in the finally clause to ensure we don't leak file descriptors.
Closing some more streams that were left open. This Bug #6136 seems to be pervasive and is going to require an extensive audit to find all of the places where streams are not closed properly.
Refactor to use IOUtils.closeQuietly() which handles nulls and streams that are already closed.
Closing FileOutputStream handles so that the OS limits on filehandles are not exceeded.
support a "force replication delete all action" during replication. This is used when we want Metacat to remove the content from the other target replicas because the DataONE delete() action was called (more powerful than just "archive").
use an independent ISet<SystemMetadata> structure to communicate objects that should be indexed by metacat-index. https://projects.ecoinformatics.org/ecoinfo/issues/5943
add space to prevent syntax error when additional clause is appended. https://projects.ecoinformatics.org/ecoinfo/issues/5929.
CHange replication 'update' query to use a LEFT JOIN so that the performance of the replication update action is improved, which had been causing an HTTP timeout for large metacat installations. See https://projects.ecoinformatics.org/ecoinfo/issues/5929.
include xml_revisions.do not allow removal of server_location = 1 documents (these are not replicas).https://redmine.dataone.org/issues/3539
move DocInfo parsing into utilities project so that it can be used by Morpho as well as Metacat.http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5737
use correct docid format when checking for existing mappings.
use CDATA for docname field in docInfo so that XML parser ignores the content that can contain characters like "&
use SchemaLocationResolver to fetch remote entries for the xml_catalog -- we want to be able to fetch included xsd files as well as use any error handling it provides for checking the schemas.
create docid-guid mapping during replication if it does not exist. we were [incorrectly] assuming that there would be SM coming with the document info that would fill this information in, but for traditional non-MN Metacat deployments there is no SM to provide a mapping. In this case we use the docid as the guid.
stream the replication "update" response rather than building up a complete list in a stringbuffer. prompted by findings on t he CN: https://redmine.dataone.org/issues/3141
remove unused "dataonelogger"
remove possibility for infinite loop in case data replication is not configured for the server and a data file is encountered (yikes!)
added logging debug statements to see where the replication timeout might be occurring.
only look up the client timeout property once, not every time we make a callhttps://redmine.dataone.org/issues/3078
configurable replication client timeouthttps://redmine.dataone.org/issues/3078
stack trace the HZ put exception during CN-CN replication
additional debugging statements for CONCURRENT_MAP_PUT error during CN-CN replication.
instead of generating SM and ORE maps during dataone configuration/MN registration, moved this all to the replication admin screen where we can target generation for specific nodes. That way it's more controlled as to when and where we generate DataONE required content....
add "Generate System Metadata" button to the replication server list display. When clicked, we generate SM for records belonging to that source server. This is only enabled when DataONE has been configured.https://redmine.dataone.org/issues/2762
optionally remove the document/data file from the filesystem completely when 'deleting' it.https://redmine.dataone.org/issues/2677
add a parameter for optionally writing EML-embedded access control rules to the Metacat DB.https://redmine.dataone.org/issues/2584https://redmine.dataone.org/issues/2583
band-aid for CN-CN replication permOrder issue when access control is embedded in EML and the system metadata is replicated before the EML. we just log the inconsistency and allow the insert to succeed https://redmine.dataone.org/issues/2583
check whether mapping (
process systemMetadata from the docInfo string before writing to the database so that we guarantee guid-docid mapping exists before attempting to look it up.
remove flag for independent system metadata replication -- these entries are replicated along with the data/metadata objects or via hazelcast when the actual object is not on the server.
only create guid->docid mapping during metadata replication if it does not already existhttp://bugzilla.ecoinformatics.org/show_bug.cgi?id=5520
do not treat access change as an update -- it should not attempt to retrieve the contents of the objecthttp://bugzilla.ecoinformatics.org/show_bug.cgi?id=5520
only create guid->docid mapping during data replication if it does not already existhttp://bugzilla.ecoinformatics.org/show_bug.cgi?id=5520
process system metadata before access rules (access control is now driven by GUID so the mapping needs to be there)
use shared method for looking up "docInfo" map -- both in Metacat replication and in D1 system metadata generation
replication control panel now fully implemented as an admin configuration screenhttp://bugzilla.ecoinformatics.org/show_bug.cgi?id=5528
move replication configuration actions to the admin servlet and out of the replication servlethttp://bugzilla.ecoinformatics.org/show_bug.cgi?id=5528
save SystemMetadata when replicating data and metadata -- this way if/when the node decides to be a DataONE MN it already has the information needed for each object
get server param only when it is expected
check replication table (not keystore) for trusted server host name match
started replication unit test
add note about alternative methods for getting cert/key
use DateTimeMarshaller for all replication date transfers
print the stacktrace when there is an error -- debuggin!
use SSL to get content from stream
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5527
skip verification -- remaining TODO
verify certificate
inspect keystore entries for matching client certificate
lookup the correct property for keystore file
use HttpClient to set up SSL connection when doing replication calls -- this will use the server's configured certificate as the client certificate on the request. The server it is calling can then inspect that certificate and decide whether or not it trusts the caller.
check client-provided certificate when servicing ReplicationServlet requests.
add User-Agent logging to support D1 requirements
include SystemMetadata when replicating data and metadata documents -- this allows us to establish the guid-to-docid mapping that is crucial for being able to read the replicated document by guid (d1 api)
do not send <systemMetadata> with the <docInfo> replication information - this is handled by the Hazelcast shared map
rely on Hazelcast to store the SystemMetadata locally for the node. Entry event listeners store the shared system metadata on their local node when alerted. TODO: remove old replication code that included system metadata xml when replicating scimeta and data
remove ServiceTypeUtil - replace with TypeMarshaller
use new "v1" types from DataONE
use correct log name for the class
add option for replicating system metadata (dataone)https://redmine.dataone.org/issues/1626
force replication for newly-registered system metadata
Merged in the D1_0_6_2_BRANCH changes that include the transition from ObjectFormat calls to ObjectFormatCache calls.
include System Metadata forced replication - just need to figure out when to call it!
handle timed replication of system metadata. there are still a few outstanding issues: -track server location of system metadata-only entries-replication policy flag for system metadata-only entries?-locking for replicated entries?-forced replication of entries
transfer full System Metadata (as XML) during document and data replication
-remove system metadata guid -> local id mapping (there is no document for system metadata now)-include system metadata elements when replicating data objects (TODO: transfer all system metadata structures with the docinfo request).TODO: remove docid+rev from the systemMetadata table definition
do not use XML files for storing SystemMetadata - use DB tables only.
use update method to update the mapping between local and guid (d1) when we get a force replication request that is an "update