Metacat

Add a comment.

Removing xml_access table records now is in the removing the pid from hazelcast. So remove the dupicated code.

Change some debug statments from debug to trace.

Add the feature to support the noNamespaceSchemalLocation and refactory the code.

Add a new constant for no-namespace schemas.

Merge changes from the 2.6.0 release.

If a format id in the system metadata is registered in the xml_catalog table, we will use the schema location for the format id to validate the xml instance;
otherwise, we will use our previous way.

During the replication, the remote content will be saved without alteration.

allow per-document reindexing to be initiated by any user (to support third-party annotations)

Write the input stream into the file system without alteration in dataone create and update methods.

change the way to delete the solr index of a resource map.

Add the code to handle the delete of the resource map.

Make the delete method work.

add support for v2 DataONE API.

recursively submit obsoleted objects for indexing when instructed. https://projects.ecoinformatics.org/ecoinfo/issues/6424

sync pids of <distribution><online> data objects with CN when their access rules change in EML 2.0.* <additionalMetadata>

Sync access policy between MN -> CN when access rules are updated in EML 2.1+ for data object

Unify solr indexing with an IndexTask that is added to the queue -- allows us to send more than just the systemMetadata to the indexer. Initially this is for READ event counts for each document. https://projects.ecoinformatics.org/ecoinfo/issues/6346

On changes to system metadata in CNodeService and DocumentImpl, increment the serialVersion.

Refactor to use IOUtils.closeQuietly() which handles nulls and streams that are already closed.

Added close() to finally block for readFromFileSystem() call.

Closing FileOutputStream handles so that the OS limits on filehandles are not exceeded.

If the pathquery engine is disabled, the xml path index queue will be disabled as well.

support a "force replication delete all action" during replication. This is used when we want Metacat to remove the content from the other target replicas because the DataONE delete() action was called (more powerful than just "archive").

do not use tmp file to return an inputstream on read() operations - just read from the file we already have. https://projects.ecoinformatics.org/ecoinfo/issues/6009

use an independent ISet<SystemMetadata> structure to communicate objects that should be indexed by metacat-index. https://projects.ecoinformatics.org/ecoinfo/issues/5943

remove indexing task from the queue when we are updating the document

move DocInfo parsing into utilities project so that it can be used by Morpho as well as Metacat.
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5737

rollback the delete() when there is an error performing part of it -- don't want to end up with partial delete.

use Identifier object not String when retrieving SM from the HZ map to set archived during delete()

remove document from the indexing queue when delete is called. http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5750

mark documents as archived=true when they are deleted using the Metacat API.
https://redmine.dataone.org/issues/3406

correct the number of prepared statement parameters when inserting to xml_revisions table.
Errors like the following were showing in the replication log file:
knb 20120831-19:42:38: [ERROR]: DocumentImpl.writeReplication - Failed to create access rule for package: john.15950.1 because The column index is out of range: 12, number of columns: 11. [ReplicationLogging]

when updating a document on a remote server, we still need to use the previous docid to check that the user has permissions to do so (rather than the new id that is obsoleting the old id). This was discovered by M Servilla at LTER.

[optionally] do not archive the xml_documents and xml_nodes to *_revisions when 'deleting' a document. This will effectively guarantee that the document/data cannot be retrieved after delete.
NOTE: D1 system metadata will persist (for now) so that the ID cannot be reused with the DataONE API but Metacat calls may allow the ID to be reused -- may need to reconsider this behavior....

optionally remove the document/data file from the filesystem completely when 'deleting' it.
https://redmine.dataone.org/issues/2677

add a parameter for optionally writing EML-embedded access control rules to the Metacat DB.
https://redmine.dataone.org/issues/2584
https://redmine.dataone.org/issues/2583

check previous revision for permissions to update (includes data described by EML)

refactor Metacat access handling to be on a per-revision basis so that it more closely aligns with the DataONE approach
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5560

uses prepared statement instead of plain old statement.
deprecated the DBConnection.createStatement() method to discourage direct parameter value use in favor of parameter binding.
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5527

http://bugzilla.ecoinformatics.org/show_bug.cgi?id=5527

include clearer error message when UPDATE action is requested on a replicated document and we fail to successfully get a lock from the source Metacat server
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=4907

do not delete the access rules when we "archive" the document on "delete" (commented out for now)

remove System.out statements in favor of logging

use the jaxb date parser for ISO 8601 formats. the numeric and date node values are now calculated after the document has been successfully inserted in the db so any sql exceptions do not prevent the raw node data from being saved.
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=2084

add support for temporal element query in pathquery
http://bugzilla.ecoinformatics.org/show_bug.cgi?id=2084

DocumentImpl.delete() now throws finer grained exceptions (not a general exception). Consequently, the classes that call it have been updated to handle the thrown exceptions, including CrudService, ReplicationHandler, and ReplicationService.

use detected XML encoding when reading/writing files
use UTF-8 as default when performing queries in the DB (assume DB is using UTF-8)
remove as many PrintWriters (uses system default character encoding only) as possible and construct OutputStreamWriters where explicit encoding can be given....

add support for EML 2.1.1

made delete serialize the identifier

fixed a bunch of small errors, did some reformatting, and fixed a bug that I thought was fixed last week

Fixed spelling error.

Correct log warning message - not "Account" but "Accession" number

removed system.outs

amost have update working. still need to get unit test squared away.

refactored XMLSchemaService to not have static methods. made the CrudServiceTest more robust.

removed system.outs

fixed schema location bug. the dataone schemas are now correclty found

removed CrudService dependency on servlet params. CrudService is now a singleton. I'm getting an error from metacat saying it can't find teh systemmetadata schema, even though it is, in fact, registered with metacat. need to identify why this is happening.

Modifications to support the DataONE service API version 0.1.0. For DataONE, the get() and
create() services are partially complete. Several more functions and checks need to be added to
create() before it is viable. This DataONE support is not complete, and the current support breaks the MetacatRestClientTest for the time being (this client will eventually be removed).

Merge 1.9.2 changes back into the trunk

Modified readFromMetacat() to pass most exceptions up the call stack, which
allows creation of new entry points for calling reads. Still need to
continue factoring out the HTTPServletResponse that is passed in in order to
make entrypoints that are not servlet based possible. Problem now is in...

Beef up log messages

do a quick retry if building path index fails with a SQL error

Make sure buildIndex throws an exception if it has a sqlexception. That way the indexing object will be added to the indexing queue and reprocessed.

Pass the doc xml as a string to docImpl.write and writeRepication. This is so a reader can be create for the parsing and for the write to disk. Also created a db access class for xml query result deletion.

Fix the bug of http://bugzilla.ecoinformatics.org/show_bug.cgi?id=4645. handleGetRevisionAndDocTypeAction will search both xml_documents and xml_revisions table.
It also changed some constrain in AccessionNumber when user update a document, of which all previous versions are in xml_revisions table.

Change log levels where appropriate and add class/method name to output.

beef up error log messages

special handling for RDF namespace documents (semtools project)

Move access control source to it's own directory.

Change location of PropertyService to properties directory

Move document specific utilities to DocumentUtil from MetacatUtil. This makes it easier to define a layer between the core metacat services and the rest of the code.

Create database and shared directories for database management code and shared code respectively.

Add archival read funtionality (jar/kar/war files)

Introduce replication user. Use the fileutil writer methods instead of writing directly.

Beef up exception handling from file utilities. Move UtilException to MetacatUtilException to eliminate conflict with similar exception in utility package.

Do not read or write zero length documents to/from disk

Format indexPaths in metacat.properties. Remove from build.properties and build.xml. Move indexPath list getter from MetacatUtil to SystemUtil.

Fix typos in error messages.

Renamed MetaCatUtil to MetacatUtil

move readerToString function to StringUtil class in utilities module.

Add debug statements

Rename LDAPUtil to AuthUtil

Move the code to write metadata to disk into documentImpl

Add some generics typing. Separate the code that strips inline data from document files to have a different strategy for 2.0.X versus 2.1.X documents.

Look for schemaLocations in the document while initializing parser. If full schema validation is turned on in metacat.properties, and at least one schema is not registered locally, then turn on full schema validation in the parser.

Fix workgroup description

add closing </distribution> tag to regex for inline data.

Created a method in DocumentImpl to read a document file from disk. Created a support method to strip inline data for users that don't have read access.

Move the DBAdaptor accessor into a DatabaseService class

qualify xml and eml properties with an xml. prefix

Continue to qualify property names

Fully qualify spatial and replication properties

Use qualified properties

Add sql debug statements

Append context url onto system id instead of server url.

Merge 1.9 changes into Head

fixed bug 3403

Remove an obsolete private method getPartNodeRecordList.