do not subset the list for MS generation testing -- at least not as the default in svn!
use IdMan method to find docids that do not already have system metadata records -- this lets us re-run without re computing system metadata for every entry (in case the process is interrupted). I haven' been using this option because I wanted to continually regenerate all SM for everything in my test DBs, but we are so close to release that I want to get this in there.
for testing: limit and randomize the docs to generate metadata for
FOR TESTING ONLY: limit number of records to 100 so that we can get an estimate
refactor D1-specific upgrade utilities into their own package
do not wait for SM generation to complete during the upgrade -- this way the web UI wont hang for days. the process sets a metacat property when it is complete.
do not shutdown hazelcast -- it needs to be running after the upgrade process so that Metacat actually works.I think the newer version of HZ makes it so the threads are all released as needed.http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5572
upgrade to hazelcast 1.9.4.6 so that threadpools are released when not needed (http://code.google.com/p/hazelcast/issues/detail?id=765).include ant target to run a specific main class (mostly for debugging)
multithreaded implementation for processing docids for system metadata generation.need to investigate ant/junit running that deadlocks hazelcast (config?)
Use the Collections class from java.util.
-generate system meta for all docids, even those not originating on the server (replicas from the past)-generate ORE docs and download remote data only for those documents that originated on this server being upgraded.http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5522
refactor generate system meta loop to the factory class -- to be reused in sysmeta and ORE generationhttp://bugzilla.ecoinformatics.org/show_bug.cgi?id=5522
do NOT generate ORE maps or download data when we do the initial System Metadata generation -- this is deferred until D1 registration.
make more generic so that a custom list of IDs can be passed in.
insert OR update system metadata -- no need to do an update right after initial insert...
In IdentifierManager.updateSystemMetadata(), add a check for invalid system metadata (fields that throw a NullPointerException on access) to ensure that system metadata is populated correctly. Updated calling classes to handle the exception.
download remote data and save locally when it is referenced by an EML package, then include it in the ORE map.http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5522
sort the docids so that "old" revisions are processed before newer ones
generating ORE maps and creating/updating system metadata now. There are some Permission conversion issues to be worked out yet
optionally include ORE generation/insertion into Metacat when generating SystemMetadatahttps://redmine.dataone.org/issues/2056
refactor SystemMetadata creation into separate class from the MetacatHandler -- this will be shared by upgrade code and normal metacat api.
include all document revisions when generating "missing" system metadataTODO: revision graph captured in obsoletes/obsoletedBy
simplify SystemMetadata generation -- will be done during Metacat upgrade for D1 features/support.
Update classes to use the DataONE 0.6.4 schema and types. Major changes involve using BigInteger vs long in SystemMetadata.size, and using ObjectFormatIdentifier rather than Object format.
use new "v1" types from DataONE
remove CrudService -- replaced by MNodeService and CNodeService
organize imports so that it is clearer what dependencies exist on the D1 jars
-remove system metadata guid -> local id mapping (there is no document for system metadata now)-include system metadata elements when replicating data objects (TODO: transfer all system metadata structures with the docinfo request).TODO: remove docid+rev from the systemMetadata table definition
do not use XML files for storing SystemMetadata - use DB tables only.