Remove null field tests in the IdentifierManager class. Schema-level required fields are checked on serialization/deserialization using JibX during the REST resource handler classes. Other required fields are checked in MNodeService and CNodeService, higher in the stack.
In IdentifierManager.updateSystemMetadata(), add a check for invalid system metadata (fields that throw a NullPointerException on access) to ensure that system metadata is populated correctly. Updated calling classes to handle the exception.
In IdentifierManager, throw SQLExceptions rather than just logging them, and let them be handled higher up in the stack.
update authoritative member node id when we change it (reconfiguration) and when we initially register as a MN with the CN.
When the requested count in a call to listObjects() is 0, return an empty object list, not a full one. Fixes https://redmine.dataone.org/issues/2122
Minor formatting for querySystemMetadata().
Change the query semantics such that we implement the MN.listObjects() where the lower datetime bound is inclusive (greater than or equal to" and the upper datetime bound in exclusive (less than). This allows easier paging in client applications.
include 'archived' system metadata element in backing DB store
use correct "rev" column in xml_revisions table
refactor Metacat access handling to be on a per-revision basis so that it more closely aligns with the DataONE approachhttp://bugzilla.ecoinformatics.org/show_bug.cgi?id=5560
for "all" permission, return a list of READ, WRITE, CHANGE_PERMISSION
look up access policy by guid or local idTODO: resolve the Metacat/EML "all" permission as it relates to DataONE (there is only READ, WRITE, CHANGE_PERMISSION). for now I am using CHANGE_PERMISSION when it is a Metacat "all"
optionally include ORE generation/insertion into Metacat when generating SystemMetadatahttps://redmine.dataone.org/issues/2056
use shared method for looking up "docInfo" map -- both in Metacat replication and in D1 system metadata generation
include all document revisions when generating "missing" system metadataTODO: revision graph captured in obsoletes/obsoletedBy
delete system metadata when MN.delete() is called.
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5527
move the DataONE 1.0.0-SNAPSHOT
Fix a data-typing issue when pulling replicationAllowed from postgres. Use getBoolean(), not getString().
IdentifierManager.getSystemMetadata() was missing the number_replicas and replication_allowed fields when building a ReplicationPolicy section of a SystemMetadata document being returned. Add in these two attributes.
-use whatever object format id is stored in the DB-only set originating node and authoritative node if we have non-null values for them
skip Objects that have null format ids are invalid checksums
more stringent NP checking when saving replication policy
lookup latest system metadata update date for use in synchronizing CN-CN when an offline nodes comes back online
do not throw exception when checking for system metadata - boolean return is good.use ReplicationStatus.valueOf() instead of convert()
treat access rules atomically - do not group them together otherwise the intent is subverted
check for null pointers when adding system metadata/creating records during cn.create()
do not add obsoletes and obsoletedBy elements if their values are null (jibx chokes on the emptiness)
fixed logic wrt localID and docid. Implemented new method in IdentifierManager to getAllGUIDs from identifier table for implementation of loadAllKeys in ObjectPathMapLoader.
changes for schema update (d1_common)
Update classes to use the DataONE 0.6.4 schema and types. Major changes involve using BigInteger vs long in SystemMetadata.size, and using ObjectFormatIdentifier rather than Object format.
Fix bug in updateSystemMetadataFields() where guid is set from the wrong parameter in the prepared SQL statement.
remove ORE mapping from system metadata
include obsoletes and obsoletedBy for system metadatarepurpose the "provenance" table as the ORE mapping table for system metadata
use correct enum method
use new "v1" types from DataONE
use objectFormatIdentifier for listObjects()remove provisional system metadata indicator - Metacat will not implement reserveIdentifier()
Reverting the resultset order in querySystemMetadata(). Thanks Ben.
Because of the new 'provisional' column, the resultset field order in querySystemMetadata() was out of order. Changed the order to reflect the new table column order.
allow very minimal system metadata for provisional entries (CN.reserveIdentifier)
handle null values from DB better when querying system metadata
Fixed problems in IdentifierManager asSystemMetadata(), getSystemMetadata(), and querySystemMetadata() that were using ObjectFormat.toString() rather than ObjectFormat.getFmtid().getValue(). Metacat was storing the string memory pointer rather than the actual format string in the systemmetadata table.
do not delete if the sysmeta doesn't exist
allow for provisional SystemMetadata records (provisional=true)
Updated CNCoreImpl to implement listFormats() and getFormat(), and changed calls to ObjectFormatCache in IdentifierManager, MetacatHandler to call getInstance(). Removed the ObjectFormatService registration from MetaCatServlet since it is replaced by CNCoreImpl.
When calling SystemMetadata.getObjectFormat(), return the value of the ObjectFormatIdentifier rather than ObjectFormat.toString() (which no longer returns the fmtid string).
Merged in the D1_0_6_2_BRANCH changes that include the transition from ObjectFormat calls to ObjectFormatCache calls.
check system metadata for the id as well (in cases when we only have system metadata)
include GUID column for xml_access and related methods for storing/retrieving access rules
handle timed replication of system metadata. there are still a few outstanding issues: -track server location of system metadata-only entries-replication policy flag for system metadata-only entries?-locking for replicated entries?-forced replication of entries
read and write D1 access policy rules from metacat xml_access table.still TBD: which mechanism takes precedence when there are systemMetadata access rules and EML access rules and other access rules?
persist system metadata replication policy and status using db tables
do not look in systemMetadata for a docid->guid mapping
-remove system metadata guid -> local id mapping (there is no document for system metadata now)-include system metadata elements when replicating data objects (TODO: transfer all system metadata structures with the docinfo request).TODO: remove docid+rev from the systemMetadata table definition
do not use XML files for storing SystemMetadata - use DB tables only.
In order to sync up with DataONE 0.6.1 changes, I'm backing out ObjectFormatService changes temporarily in Metacat. Most functionality will be rolled back in using the DataONE 0.6.2 tag, but some methods in ObjectFormatService (such as getListFromDisk()) will be moved into d1_libclient_java.
Changes in the DataONE ObjectFormat class deprecate the convert() method, and we're now using Metacat's ObjectFormatService to look up object format attributes. The following changes replace ObjectFormat.convert() with ObjectFormatService.getFormat() in several classes....
use update method to update the mapping between local and guid (d1) when we get a force replication request that is an "update
Modified IdentifierManager.getDocumentInfo() to include the docid in the returned hash map, since it is useful to be able to obtain the docid and rev separately from a given fullDocidWithRev (e.g. test.1.1).
fixed bug where the wrong checksum alg got written to the db
adding more debuggin and fixing bug with systemmetadata
fixed replication bug where systemmetadata was not getting procssed correctly
think I fixed the connection problem. one connection in IdentifierManager was being leaked. added more debug info in case it happens again
fixed typo that prevented replication
fixed problem with count in listObjects()
fixed bugs in listObjects
few bug fixes for listObjects
added code to do database query for listObjects
adding fields for additional system metadata info
fixes for creating SM for legacy docs
fixing generateSystemMetadata
still working on getting guid problems fixed with replication
fixed identifier manager problem with getting guid for systememtadaa doc
fixing problems with replication and systemmetadata
fixed more access control problems. also fixed extraneous white space that was getting put into the listobject output stream
fixed problems with access control on sysmeta
fixes for the access control setting for sysmeta
added functionality to set access permissions to system metadata the same as the document that it describes
fixed bug in access control where the sessionid was not correctly passed along so changes of privileges were not being made
working on getting systemmetadata creation working for legacy knb objects
working on a function to create dataone system metadata docs for legacy metacat objects that do not have SM.
removed system.outs
amost have update working. still need to get unit test squared away.
getting getSystemMetadata to work
refactoring to remove duplicate code
adding a system to track system metadata documents for dataone
added a method in IdentifierManager to get a guid from a docid and rev. added fields in the documentinfo replication document to pass the guid. now need to handle the guid and insert it into the table if its found
Added support to ResourceHandler to allow the putObject method to usearbitrary guid strings as input. These strings are examined, and if theymatch the Metacat docid format (scope.id.rev), they are used directly. Ifthe id is a string in another format, a new localId is generated based on...
Added initial support for arbitrary string identifiers in metacat (referred to as a guid). The new identifier table is used to map arbitrary string identifiers to Metacat's current docid format (referred to as the localId). Added a new IdentifierManager class to manage this table, adding new mappings as objects are added to the system. Modified the MetacatTest service to utilize this mapping table to look up a localId from a guid. IdentiferManagerTest is working with these guids now, as is the MetacatTest service get() method, but other parts of the system are unaware of them (e.g., create, update, delete operations are unaware, and no mapping is created when new documents are created). As a consequence, the MetacatRestClientTest is not working (although it also had lots of hardcoded dependencies that need to be fixed as well).