select only distinct guids (synch may have failed more than once for any given guid)https://redmine.dataone.org/issues/3539
include xml_revisions.do not allow removal of server_location = 1 documents (these are not replicas).https://redmine.dataone.org/issues/3539
class for removing failed/invalid replicas from target nodes that previously held replicated content (KNB/LTER/PISCO/etc). https://redmine.dataone.org/issues/3539
shorten the systemmetadata* table names for Oracle's 30 character limit. move version to 2.0.5. http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5717
check for null archived flag in ORE SMhttps://redmine.dataone.org/issues/3046
utility methods to update/reserialize existing ORE maps that were generated with older foresite (and included bad dateTime strings).https://redmine.dataone.org/issues/3046
include dataone.ore.downloaddata as a configurable property in case MNs (like LTER) want to have the process download externally-stored data files described in an EML data package.
instead of generating SM and ORE maps during dataone configuration/MN registration, moved this all to the replication admin screen where we can target generation for specific nodes. That way it's more controlled as to when and where we generate DataONE required content....
include all EML versions (had been only eml 2.1 for testing)
expose serverLocation parameter to run GenerateSystemMetadata for different replication parters as needed.https://redmine.dataone.org/issues/2740
only generate system metadata for original objects.https://redmine.dataone.org/issues/2721
do not download data at this point
only 2.1.0 EML docs for ORE generation right now...
handle last group of ids (oops)
use range of the list for test system metadata
use non-random list for generating system metadata in test mode
use scope prefix (knb-lter-%) for random test doc population
do not subset the list for MS generation testing -- at least not as the default in svn!
only run ORE generation for EML docs -- no need to run this for all documents (yikes!)
use IdMan method to find docids that do not already have system metadata records -- this lets us re-run without re computing system metadata for every entry (in case the process is interrupted). I haven' been using this option because I wanted to continually regenerate all SM for everything in my test DBs, but we are so close to release that I want to get this in there.
for testing: limit and randomize the docs to generate metadata for
FOR TESTING ONLY: limit number of records to 100 so that we can get an estimate
refactor D1-specific upgrade utilities into their own package
generate SystemMetadata during D1 registration (not 2.0.0 upgrade). This process runs in a thread and updates a metacat.properties value when it is complete.
use correct Collections import
do not wait for SM generation to complete during the upgrade -- this way the web UI wont hang for days. the process sets a metacat property when it is complete.
do not shutdown hazelcast -- it needs to be running after the upgrade process so that Metacat actually works.I think the newer version of HZ makes it so the threads are all released as needed.http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5572
Commenting out the parts of the upgrade script that started to refer to EXID. At this point, the registration of EZID identifiers will be done out-of-band with respect to the upgrade.
upgrade to hazelcast 1.9.4.6 so that threadpools are released when not needed (http://code.google.com/p/hazelcast/issues/detail?id=765).include ant target to run a specific main class (mostly for debugging)
multithreaded implementation for processing docids for system metadata generation.need to investigate ant/junit running that deadlocks hazelcast (config?)
only attempt to update date-like nodedata values.
include generate system metadata upgrade in the success flag
more clean up - reuse prepared statement for data update
look up nodedata values first, then update each one - trying to avoid out of memory exception.
Added DOI generation to the 2.0.0 upgrade process. To succeed, this script must be run on a fresh 2.0.0 database, or on a 1.9.5 version database, as those are the only ways to get the needed foreign keys to be marked as deferrable. The identifier conversion must be turned on by setting correct properties in metacat.properties. See the comments in GenerateGlobalIdentifiers for details. By default, conversion is set to false in the properties file. If you want to convert an instance to use DOIs, be sure to set metacat.properties up BEFORE running through the Metacat configuration and database upgrade.
Use the Collections class from java.util.
-generate system meta for all docids, even those not originating on the server (replicas from the past)-generate ORE docs and download remote data only for those documents that originated on this server being upgraded.http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5522
refactor generate system meta loop to the factory class -- to be reused in sysmeta and ORE generationhttp://bugzilla.ecoinformatics.org/show_bug.cgi?id=5522
do NOT generate ORE maps or download data when we do the initial System Metadata generation -- this is deferred until D1 registration.
make more generic so that a custom list of IDs can be passed in.
insert OR update system metadata -- no need to do an update right after initial insert...
call the System Metadata generator during upgrade to 2.0.0
In IdentifierManager.updateSystemMetadata(), add a check for invalid system metadata (fields that throw a NullPointerException on access) to ensure that system metadata is populated correctly. Updated calling classes to handle the exception.
only delete replicated data files (server_location != 1)
download remote data and save locally when it is referenced by an EML package, then include it in the ORE map.http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5522
upgrade routine to purge empty replicated data files so that they can be re-replicatedhttp://bugzilla.ecoinformatics.org/show_bug.cgi?id=5536
sort the docids so that "old" revisions are processed before newer ones
close prepared statement only if not nullhttp://bugzilla.ecoinformatics.org/show_bug.cgi?id=5562
generating ORE maps and creating/updating system metadata now. There are some Permission conversion issues to be worked out yet
optionally include ORE generation/insertion into Metacat when generating SystemMetadatahttps://redmine.dataone.org/issues/2056
refactor SystemMetadata creation into separate class from the MetacatHandler -- this will be shared by upgrade code and normal metacat api.
include all document revisions when generating "missing" system metadataTODO: revision graph captured in obsoletes/obsoletedBy
simplify SystemMetadata generation -- will be done during Metacat upgrade for D1 features/support.
use logging, not system.out
change upgrade scripts/routines to use 2.0.0 version number instead of 1.10.0
make sure we close the prepared statement always
Update classes to use the DataONE 0.6.4 schema and types. Major changes involve using BigInteger vs long in SystemMetadata.size, and using ObjectFormatIdentifier rather than Object format.
use new "v1" types from DataONE
remove CrudService -- replaced by MNodeService and CNodeService
organize imports so that it is clearer what dependencies exist on the D1 jars
-remove system metadata guid -> local id mapping (there is no document for system metadata now)-include system metadata elements when replicating data objects (TODO: transfer all system metadata structures with the docinfo request).TODO: remove docid+rev from the systemMetadata table definition
do not use XML files for storing SystemMetadata - use DB tables only.
do each table separately with it's own connection - running into memory issues on dev.nceas running this.
use the jaxb date parser for ISO 8601 formats. the numeric and date node values are now calculated after the document has been successfully inserted in the db so any sql exceptions do not prevent the raw node data from being saved.http://bugzilla.ecoinformatics.org/show_bug.cgi?id=2084
add support for temporal element query in pathqueryhttp://bugzilla.ecoinformatics.org/show_bug.cgi?id=2084