Use the NodeReference object to replace the replicaStatus to restrict the listObjects method.
Fixed a bug that two columns were missing in the sql command.
Add the code to read/write the new system metadata fields - file name, media type and media type properties into the db tables.
Throw an exception if the saving system metadata fails.
Use the log to the replace system.out.print.
Add the code to get the head version.
Implemented the feature to support the identifier(both sid and pid) feature in the objectLists method.
Add the series id to the system metadata.
Modified the get method to handle the sid in the v2 api.
Change a method name to use pid rather than guid.
Add the code to get the most recent version of a sid which all pids are obsoleted.
Fixed a bug that the method getLocalId swallowed an exception incorrectly in the IdentifierManager.
1. In the identifierExists method, the code to check if the identifier exist on the serial-id fields of the system metadata table.2. Fixed a bug on the identifierExists - it considered an identifier didn't exist when the determining process threw an exception.
Add a method to check if a serial id exists.
Add a method to get the pid of the head version of a sid.
Make the deleteSystemmetadata method really roll-backable.
add support for v2 DataONE API.
Allow null SM.submitter (per schema). There were null values in cn-dev (and probably elsewhere since it is technically allowed in the schema. But with a null value, we need to have a null Subject for the SM.submitter field, not a Subject with a null getValue() return. Encountered this when testing for: https://projects.ecoinformatics.org/ecoinfo/issues/5929.
move DocInfo parsing into utilities project so that it can be used by Morpho as well as Metacat.http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5737
default replicaStatus (aka "show replicas in results") to true rather than false
add debug statements for listObject slice debugging
use dual query for query slicing - one for count, another for the actual records when requested.https://redmine.dataone.org/issues/3065
Fix a minor bug in listObjects() where total was set incorrectly when total was set incorrectly when count=0. The definition of total in the D1 architecture docs says 'The total number of entries in the source list from which the slice was extracted.' With count=0, we assume the total is the total count from the entire object store. Needs testing.
use RDBMS-specific features to limit the resultset for paging the object list -- postgres and oracle have implementations. we don''t really support mssql so I skipped that one.
look up the archived value when retrieving SystemMetadata record.https://redmine.dataone.org/issues/3405
handle /object?count=0 queries using simpler (quicker) sql https://redmine.dataone.org/issues/3065
Change the set and get methods for the replication verified date to use java.sql.Timestamp rather than java.util.Date via setTimestamp(), not setDate(). The hh:mm:ss.sss was previously getting truncated.
shorten the systemmetadata* table names for Oracle's 30 character limit. move version to 2.0.5. http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5717
order the listObjects() results by identifier to mitigate random paged resultshttps://redmine.dataone.org/issues/3065
share the same dbConnection when inserting and then updating SystemMetadata objects in the backing store.any errors encountered during the update will rollback the entire transaction and the SM record will not exist, even in part.
Do not loadAllKeys() for SystemMetadataMap when Metacat first starts up. hzIdentifiers will be populated with a simple SQL statement rather than the serial loading of every single SystemMetadata object. It will remain in synch using the usual entryXXX() methods as before....
only generate system metadata for original objects.https://redmine.dataone.org/issues/2721
include debug statements for systemMetadataReplicationStatus and systemMetadataReplicationPolicy SQL
handle case where EML access rule "permission" is not in our constrained list (EML 2.0.0 doc showed this with a "none" permission for public principal). we now omit this invalid access rule when interpreting it in system metadata -- effectively dropping that invalid access rule. "none" had been stored as a 0 in the DB xml_access table and would not have given or denied access for the document so I think it can safely be omitted for good. for example, see knb-lter-gce.101.2 with this rule:...
do not allow "Metacat-conforming" identifiers to be used. "test.1.001" is interpreted as "test.1.1" which renders "test.1.002" unusable unless a traditional Metacat "update" is used for that id/revision which contradicts the DataONE use of Identifiers that have no lexical requirements for revisions.
use Timestamp object from results, not a new Date object from that Timestamp object
check whether mapping (
Globally change the property 'dataone.memberNodeId' to 'dataone.nodeId'. This is more useful for both MNs and CNs implemented in Metacat. Also, change D1NodeService.getLogRecords() to return log entries with the actual node id rather than the IP address (looks like a cut/paste error)....
do not "lookup" object format when retrieving system metadata -- just return what we have stored as the formatId and don't [erroneously] default it to binary when there's a problem with the lookup (cache or service or otherwise).https://redmine.dataone.org/issues/2365
actually, let's set the serialVersion during the MN.create() call so that the HZ map and the backing store have the same information immediately. Also, this is how the docs specify it.http://mule1.dataone.org/ArchitectureDocs-current/design/SystemMetadata.html
if serialVersion is null, use defualt value of 0
remove ID mapping when a create()/"insert" call fails so that subsequent calls do not return an IdentifierNotUnique error. In this case it was due to invalid XML.https://redmine.dataone.org/issues/2341
ObjectFormatCache.getFormat(String formatStr) has been deprecated, and now only takes a formatId instance to get a format from the cache. It also throws Service Failure and NotImplemented, so here just set the format to application/octet-stream in any case.
refactor IdentityManager.createSystemMetadata(sm) to be insertSystemMetadata(sm) so that it is clear that this method inserts the SM object into the backing store. This differentiates it from the "generation" methods we use when we need to create SM about pre-existing objects or objects we get from non-D1 api calls.
Remove null field tests in the IdentifierManager class. Schema-level required fields are checked on serialization/deserialization using JibX during the REST resource handler classes. Other required fields are checked in MNodeService and CNodeService, higher in the stack.
In IdentifierManager.updateSystemMetadata(), add a check for invalid system metadata (fields that throw a NullPointerException on access) to ensure that system metadata is populated correctly. Updated calling classes to handle the exception.
In IdentifierManager, throw SQLExceptions rather than just logging them, and let them be handled higher up in the stack.
update authoritative member node id when we change it (reconfiguration) and when we initially register as a MN with the CN.
When the requested count in a call to listObjects() is 0, return an empty object list, not a full one. Fixes https://redmine.dataone.org/issues/2122
Minor formatting for querySystemMetadata().
Change the query semantics such that we implement the MN.listObjects() where the lower datetime bound is inclusive (greater than or equal to" and the upper datetime bound in exclusive (less than). This allows easier paging in client applications.
include 'archived' system metadata element in backing DB store
use correct "rev" column in xml_revisions table
refactor Metacat access handling to be on a per-revision basis so that it more closely aligns with the DataONE approachhttp://bugzilla.ecoinformatics.org/show_bug.cgi?id=5560
for "all" permission, return a list of READ, WRITE, CHANGE_PERMISSION
look up access policy by guid or local idTODO: resolve the Metacat/EML "all" permission as it relates to DataONE (there is only READ, WRITE, CHANGE_PERMISSION). for now I am using CHANGE_PERMISSION when it is a Metacat "all"
optionally include ORE generation/insertion into Metacat when generating SystemMetadatahttps://redmine.dataone.org/issues/2056
use shared method for looking up "docInfo" map -- both in Metacat replication and in D1 system metadata generation
include all document revisions when generating "missing" system metadataTODO: revision graph captured in obsoletes/obsoletedBy
delete system metadata when MN.delete() is called.
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5527
move the DataONE 1.0.0-SNAPSHOT
Fix a data-typing issue when pulling replicationAllowed from postgres. Use getBoolean(), not getString().
IdentifierManager.getSystemMetadata() was missing the number_replicas and replication_allowed fields when building a ReplicationPolicy section of a SystemMetadata document being returned. Add in these two attributes.
-use whatever object format id is stored in the DB-only set originating node and authoritative node if we have non-null values for them
skip Objects that have null format ids are invalid checksums
more stringent NP checking when saving replication policy
lookup latest system metadata update date for use in synchronizing CN-CN when an offline nodes comes back online
do not throw exception when checking for system metadata - boolean return is good.use ReplicationStatus.valueOf() instead of convert()
treat access rules atomically - do not group them together otherwise the intent is subverted
check for null pointers when adding system metadata/creating records during cn.create()
do not add obsoletes and obsoletedBy elements if their values are null (jibx chokes on the emptiness)
fixed logic wrt localID and docid. Implemented new method in IdentifierManager to getAllGUIDs from identifier table for implementation of loadAllKeys in ObjectPathMapLoader.
changes for schema update (d1_common)
Update classes to use the DataONE 0.6.4 schema and types. Major changes involve using BigInteger vs long in SystemMetadata.size, and using ObjectFormatIdentifier rather than Object format.
Fix bug in updateSystemMetadataFields() where guid is set from the wrong parameter in the prepared SQL statement.
remove ORE mapping from system metadata
include obsoletes and obsoletedBy for system metadatarepurpose the "provenance" table as the ORE mapping table for system metadata
use correct enum method
use new "v1" types from DataONE
use objectFormatIdentifier for listObjects()remove provisional system metadata indicator - Metacat will not implement reserveIdentifier()
Reverting the resultset order in querySystemMetadata(). Thanks Ben.
Because of the new 'provisional' column, the resultset field order in querySystemMetadata() was out of order. Changed the order to reflect the new table column order.
allow very minimal system metadata for provisional entries (CN.reserveIdentifier)
handle null values from DB better when querying system metadata
Fixed problems in IdentifierManager asSystemMetadata(), getSystemMetadata(), and querySystemMetadata() that were using ObjectFormat.toString() rather than ObjectFormat.getFmtid().getValue(). Metacat was storing the string memory pointer rather than the actual format string in the systemmetadata table.
do not delete if the sysmeta doesn't exist
allow for provisional SystemMetadata records (provisional=true)
Updated CNCoreImpl to implement listFormats() and getFormat(), and changed calls to ObjectFormatCache in IdentifierManager, MetacatHandler to call getInstance(). Removed the ObjectFormatService registration from MetaCatServlet since it is replaced by CNCoreImpl.
When calling SystemMetadata.getObjectFormat(), return the value of the ObjectFormatIdentifier rather than ObjectFormat.toString() (which no longer returns the fmtid string).
Merged in the D1_0_6_2_BRANCH changes that include the transition from ObjectFormat calls to ObjectFormatCache calls.
check system metadata for the id as well (in cases when we only have system metadata)
include GUID column for xml_access and related methods for storing/retrieving access rules
handle timed replication of system metadata. there are still a few outstanding issues: -track server location of system metadata-only entries-replication policy flag for system metadata-only entries?-locking for replicated entries?-forced replication of entries
read and write D1 access policy rules from metacat xml_access table.still TBD: which mechanism takes precedence when there are systemMetadata access rules and EML access rules and other access rules?
persist system metadata replication policy and status using db tables
do not look in systemMetadata for a docid->guid mapping
-remove system metadata guid -> local id mapping (there is no document for system metadata now)-include system metadata elements when replicating data objects (TODO: transfer all system metadata structures with the docinfo request).TODO: remove docid+rev from the systemMetadata table definition
do not use XML files for storing SystemMetadata - use DB tables only.
In order to sync up with DataONE 0.6.1 changes, I'm backing out ObjectFormatService changes temporarily in Metacat. Most functionality will be rolled back in using the DataONE 0.6.2 tag, but some methods in ObjectFormatService (such as getListFromDisk()) will be moved into d1_libclient_java.