only lock the missing pid event if we know we have it locally to contribute.https://redmine.dataone.org/issues/3117
Add locking to the itemAdded() method so ideally only one CN will respond to the request for a 'wanted' pid from the cluster. The lock is on a string, not the pid, and so won't conflict with system metadata locking. The string is based on the pid, with "missing-" as a prefix.
only publish to the missing pid "wanted list" when resynching system metadata. we were seeing redundant entry added/updated events when looking up the shared systemmetadata first.
print the missing pid count, not the total shared pid count so we know how many will be processed.
change the system metadata resynch approach: nodes will publish PIDs that they are missing after inspecting the shared identifier set. other nodes will be listening for the "wanted" pids and will put their local copy of SystemMetadata on the shared SM map. This should dramatically decrease the hazelcast chatter during a resynch and targets only the pids that are missing from any of the various nodes.
logging for processing identifier set on restart.
remove possibility for infinite loop in case data replication is not configured for the server and a data file is encountered (yikes!)
added logging debug statements to see where the replication timeout might be occurring.
use correct EZID account names for the three different nodes.https://redmine.dataone.org/issues/2815
align the final column headers with the datacite schema, as applicable.https://redmine.dataone.org/issues/2815
add block for finding and updating records that should be marked as archived.https://redmine.dataone.org/issues/3109
use DataCite isNewVersionOf/isPreviousVersionOf for revision history
check for null archived flag in ORE SMhttps://redmine.dataone.org/issues/3046
check if the caller is the Node admin (the member node calling itself) as well as the existing check for the CN calling the service. Both of those callers should be given full admin rights.
not every EML file has an ORE datapackage descriptor -- join only to those when setting the resourceMapId
correctly use document revision for object format and resource map joins.
use local Set processing to determine which pids (if any) should be contributed to the shared set by this node during the resync. Should save time rather than checking each and every pid against the shared set.
move the hzIdentifiers initialization into the resync thread so that it does not affect start up time. cleaned up unused methods and superfluous code.
use correct children of 'publisher' element
only load local pids into hzIdentifiers if t hey do not already exist in the shared set. increase logging severity and detail of messages emitted during this process to get a better sense of what is taking so long.
utility methods to update/reserialize existing ORE maps that were generated with older foresite (and included bad dateTime strings).https://redmine.dataone.org/issues/3046
include the resourceMapId for the metadata objects, not just the data files.
updated LDAP dump and corrected missing entries that had been removed from LDAP.
On the coordinating Nodes, we often get McdbDocNotFoundExceptions for data (doctype == 'BIN') documents because they are not synchronized to the CNs. Change the logging to only print the stack trace during load() and loadAll() when log debug is enabled.
check for invalid (!) pids. thanks, M. Reyes for catching thishttps://redmine.dataone.org/issues/3047
only look up the client timeout property once, not every time we make a callhttps://redmine.dataone.org/issues/3078
improve content type handling during the get() callshttps://redmine.dataone.org/issues/3070
check for whitespace in identifiers during create() and update()https://redmine.dataone.org/issues/3047
configurable replication client timeouthttps://redmine.dataone.org/issues/3078
order the listObjects() results by identifier to mitigate random paged resultshttps://redmine.dataone.org/issues/3065
correct the parameter/value setting in the prepared statements for retrieving log information.
use docid, not the guid when returning the accesscontrol block
handle null givenNames from the LDAP dump.
make sure we only get the publisher text content (not attribute value)
DOI registration:-include more revision history based on the identifier table not just the generated SM metadata-include ecogrid data urls for revisions (long query in xml_nodes_revisions table)
prep for 2.0.2 release by updating the version numbers.
include dataone.ore.downloaddata as a configurable property in case MNs (like LTER) want to have the process download externally-stored data files described in an EML data package.
set date SM modified when we are setting obsoletes/obsoletedBy/archived values. This way the CN can actualy pick up the changes in revision history.
update creator and publisher using LDAP dump. unfortunately LDAP has shifted over the years and not all identities are still active in LDAP...but we did get quite a few creator names updated!https://redmine.dataone.org/issues/2815
log error when looking up non-existent local SM rather than completely bombing out of the resynch thread.
include parameter for deleting system metadata records (or not). Intending to also use this for https://redmine.dataone.org/issues/3055
look up docid using mapped guid when checking permission on described data fileAddresses: http://support.nceas.ucsb.edu/rt/Ticket/Display.html?id=7490
function/procedure for removing all content related to a PID from the DB.https://redmine.dataone.org/issues/3037
use docid (not guid) when instantiating the PermissionController. Was getting an error with DOI-ified identifier and the metacat getaccesscontrol action:https://knb.ecoinformatics.org/knb/metacat?action=getaccesscontrol&docid=Collinge.3.28<error>AccessControlForSingleFile.getACL() - MCDB error when getting ACL: No guid registered for docid doi:10.5063/AA/Collinge.3.28...
save point - adding more columns for access, data packaging, revision historyhttps://redmine.dataone.org/issues/2815
script to find and update missing SystemMetadata revision history. https://redmine.dataone.org/issues/2938
update the table to indicate which DOI account we are targetinghttps://redmine.dataone.org/issues/2815
make sure we have non-null values where jibx serialization expects them for LogEntry
use secure Metacat context URL for D1 registrationhttps://redmine.dataone.org/issues/3030
first pass: DataONE-specific log retrieval to avoid java-based post-processing.
use production cn url for the resolve url
remove the non-doi identifiers before updating the LTER - should save time on the update.https://redmine.dataone.org/issues/2858
set archived flag (true) when we set the obsoletedBy value in the ORE system metadata
update for 2.0.1 upgrade -- scripts, docs, readme
use the localId for obsoletes/obsoletedBy ORE system metadata (https://redmine.dataone.org/issues/2964)
Print the stack trace when the MMP cannot be resolved.
report errors during XML->HTML transformhttp://bugzilla.ecoinformatics.org/show_bug.cgi?id=5618
Oops, previous commit suffered from a happy trigger finger. During deleteReplicationMetadata(), don't delete the replica on the replica Member Node. Call CN.delete() for that functionality. This call just updates sytem metadata (according to the API description).
Minor logging change.
Add debug logging to delete() to understand why we're getting InsufficientKarmaException.
use KNB identifier table (post DOI generation) to update LTER identifier table.https://redmine.dataone.org/issues/2857
Since we already have determined access via isAuthorized() and isAdminAuthorized(), act as the Metacat administrator during calls to DocumentImpl.delete() in archive(), passing in null username and group.
restrict getLogRecrods (both MN and CN) to be called only by admin users (the CN)https://redmine.dataone.org/issues/2855
In setReplicationStatus() and UpdateReplicationMetadata(), don't allow a status state change from COMPLETED to anything other than INVALIDATED. This prevents the completed status from being overwritten due to race conditions.
script to aid in converting denyFirst rules to allowFirsthttps://redmine.dataone.org/issues/2613
use kepler account as default
use metacat.properties to specify the default checksum algorithm to use -- this way it will be easy for us to switch to whatever DataONE decrees. https://redmine.dataone.org/issues/2834
put(sm) for every pid we have a SM value for so that all members receive the entry event and can save locally.
Throw an exception when NOT allowed, not when allowed =).
ignore partition owner -- always attempt to look up form local store if we were unable to get the SM from the shared map.
do not check if this CN has a "perfect" copy of the SM identifiers -- we need any CN coming online to contribute the records that they have locally so that in the event that all three CNs have a partial view of things they all eventually share each others' SM entries.
Also get the list size, which may throw an NPE.
Only add an AccessPolicy to SystemMetadata during generation when the AccessPolicy is not empty. We've had some scenarios where IdentifierManager.getaccessPolicy() is returning an empty policy because of an empty permission list coming from the db. This was causing InvalidSystemMetadata exceptions during MN to MN replication.
push SystemMetadata entries from the CN that has them all to the shared map where other nodes may not have all entries. The CN with the complete copy only pushes SM entries that it does not own and that return as null because those are the ones that are missing on the other, non-complete CNs....
trace level log for looping over EVERY pid in the system.
meant to log the guids (source) not the pids (target)
logging for each step of shared identifiers loading.
remove pause/resume - seemed to make metacat just hang on SM retrieval. Add more logging when returned SM is null -- want to make sure it is becuase the local node "owns" the pid key even though there is no value for it.
due to hudson build issue, did not actually end up testing pause/resume -- trying that again
pause/resume was not enough. trying shutdown/restart
experiment with lifecycle pause/resume. hopefully it prevents our node from taking ownership of any keys before we are sure we have them all.
increase logging and add back in the call to saveLocally() in case the SM object has already been loaded into the shared map but before this node came back online.
no need to call saveLocally explicitly since loading from the shared store triggers that behavior locally because of the configured listeners.use an iterator over the shared identifiers in case this set is constantly changing.
make only one DB call to look up local pids - no need to do a pstmt for every single shared pid.
on init (start up) launch a synchronization thread that ensures all shared identifier entries have a corresponding local System Metadata entry.
use 'allowFirst' for access rules. We have deprecated 'denyFirst' and deny rules in Metacat as of 2.0.0
handle https-only server configuration -- must pull resources from https not http for the skins etc.
fix NPE (logMetacat object was not initialized) that was occurring during store()
stack trace the HZ put exception during CN-CN replication
additional debugging statements for CONCURRENT_MAP_PUT error during CN-CN replication.
include eml2.0.0beta4 DTD during Metacat build so that we can continue to accept (and validate) beta4 documents.This arose when testing Metacat as DataONE Coordinating Node where legacy documents are being housed in the CN.
encode '/' and ':' in the DOI used for the resolve URL
Don't set the replication status to failed for an object when it is called by a public user. Just throw the NotAuthorized exception. This prevents this node from being de-prioritized because of public calls to the method.
include revisions table in the initial temp table population.use the "first" creator listed in the EML (either org or person).use other reasonable default values as needed to fully populate the spreadsheet columnshttps://redmine.dataone.org/issues/2815
add columns: publisher and pub_date. include default values for all columns - even data files should have title.still a few todos but closer.https://redmine.dataone.org/issues/2815
script to generate DOI registration spreadsheethttps://redmine.dataone.org/issues/2815
share the same dbConnection when inserting and then updating SystemMetadata objects in the backing store.any errors encountered during the update will rollback the entire transaction and the SM record will not exist, even in part.