Metacat

Added DOI generation to the 2.0.0 upgrade process. To succeed, this script must be run on a fresh 2.0.0 database, or on a 1.9.5 version database, as those are the only ways to get the needed foreign keys to be marked as deferrable. The identifier conversion must be turned on by setting correct properties in metacat.properties. See the comments in GenerateGlobalIdentifiers for details. By default, conversion is set to false in the properties file. If you want to convert an instance to use DOIs, be sure to set metacat.properties up BEFORE running through the Metacat configuration and database upgrade.

Use the Collections class from java.util.

-generate system meta for all docids, even those not originating on the server (replicas from the past)
-generate ORE docs and download remote data only for those documents that originated on this server being upgraded.
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5522

refactor generate system meta loop to the factory class -- to be reused in sysmeta and ORE generation
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5522

do NOT generate ORE maps or download data when we do the initial System Metadata generation -- this is deferred until D1 registration.

make more generic so that a custom list of IDs can be passed in.

insert OR update system metadata -- no need to do an update right after initial insert...

call the System Metadata generator during upgrade to 2.0.0

In IdentifierManager.updateSystemMetadata(), add a check for invalid system metadata (fields that throw a NullPointerException on access) to ensure that system metadata is populated correctly. Updated calling classes to handle the exception.

only delete replicated data files (server_location != 1)

download remote data and save locally when it is referenced by an EML package, then include it in the ORE map.
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5522

upgrade routine to purge empty replicated data files so that they can be re-replicated
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5536

sort the docids so that "old" revisions are processed before newer ones

close prepared statement only if not null
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5562

generating ORE maps and creating/updating system metadata now. There are some Permission conversion issues to be worked out yet

optionally include ORE generation/insertion into Metacat when generating SystemMetadata
https://redmine.dataone.org/issues/2056

refactor SystemMetadata creation into separate class from the MetacatHandler -- this will be shared by upgrade code and normal metacat api.

include all document revisions when generating "missing" system metadata
TODO: revision graph captured in obsoletes/obsoletedBy

simplify SystemMetadata generation -- will be done during Metacat upgrade for D1 features/support.

use logging, not system.out

change upgrade scripts/routines to use 2.0.0 version number instead of 1.10.0

make sure we close the prepared statement always

Update classes to use the DataONE 0.6.4 schema and types. Major changes involve using BigInteger vs long in SystemMetadata.size, and using ObjectFormatIdentifier rather than Object format.

use new "v1" types from DataONE

remove CrudService -- replaced by MNodeService and CNodeService

organize imports so that it is clearer what dependencies exist on the D1 jars

-remove system metadata guid -> local id mapping (there is no document for system metadata now)
-include system metadata elements when replicating data objects (TODO: transfer all system metadata structures with the docinfo request).
TODO: remove docid+rev from the systemMetadata table definition

do not use XML files for storing SystemMetadata - use DB tables only.

do each table separately with it's own connection - running into memory issues on dev.nceas running this.

use the jaxb date parser for ISO 8601 formats. the numeric and date node values are now calculated after the document has been successfully inserted in the db so any sql exceptions do not prevent the raw node data from being saved.
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=2084

add support for temporal element query in pathquery
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=2084