Metacat

Update D1NodeService to reflect new ObjectFormatCache signature.

only run ORE generation for EML docs -- no need to run this for all documents (yikes!)

use IdMan method to find docids that do not already have system metadata records -- this lets us re-run without re computing system metadata for every entry (in case the process is interrupted). I haven' been using this option because I wanted to continually regenerate all SM for everything in my test DBs, but we are so close to release that I want to get this in there.

for testing: limit and randomize the docs to generate metadata for

FOR TESTING ONLY: limit number of records to 100 so that we can get an estimate

update the memberNodeId in existing system metadata only after the register/update is successful with the CN -- we can avoid unneeded SM updates in cases when the register/update fails because we gave the CN bad info that it rejects.
https://redmine.dataone.org/issues/2308

include member node id text field now that the CN is not assigning random Ids.
https://redmine.dataone.org/issues/2308

1. lookup and use the guid when processing obsoletes/obsoletedBy entries -- had previously been assuming localId==guid but now that we have introduced DOIs as part of the Metacat upgrade process, we may have DOIs for the guid that map to localIds.
2. base ORE guids on the localid of the data package they are describing and not on their DOI -- otherwise we might mash up the DOI prefix (or other id scheme that we are unaware of). By using resourceMapPreix + localId we are sure to have a valid localid and guid for the ORE map we create and add to the system

use updated authorization policies as discussed in:
https://redmine.dataone.org/issues/2277
and
http://epad.dataone.org/20120131-authn-authz-questions

refactor D1-specific upgrade utilities into their own package