remove semantic annotation proposal - moved to github: https://github.com/DataONEorg/sem-prov-design/blob/master/docs/use-cases/semantics/semantic-annotation.md
In InsertORETest: Set the format ID of the metadata object to an EML formatId so that it gets indexed correctly.
Change the d1_cn_index_processor version from 1.3.0 snapshot to 2.0.0 snapshot.
Add the code to delete the records in the xml_accesssubtree table.
use configured auth.base rather than hard-coded dc=ecoinformatics,dc=org. https://projects.ecoinformatics.org/ecoinfo/issues/6592
Add a code to check if the pathquery engine is enabled in the checkIndexPaths method.
Adjust the number of schema fields since new ones were added.
When indexing annotations from RDFs, use the doc id to access the system metadata, not the model name since they are not always the same.
Add PROV relationships to the Solr schema. Populate the fields using the RdfXmlSubprocessor
Add wasDerivedFrom field to the Solr schema and use Sparql query to retrieve the value from the RDF
Replace the /u00A0 character encoding with space character instead since /u00A0 displays literally in browsers
Add a test class that inserts an ORE with PROV relationships
use mock CN for testing metacat implementations
remove unused tests
comment out myproxy servlets. https://redmine.dataone.org/issues/5742
convert v2 SM to v1 SM for the v1 service call response
Separate the target and source version for the java compilation.
update to use v2 types for indexing
Add target and source attribute to harvester, client and compile-lsid beside the target compile.
Add the attribute target="1.6" when it compiles the metacat code. The metacat.jar file can run in java 1.6 even though it was compiled in java 1.7.
For the existing uidnumber, we decrease the size of vector for sorting.
Login automatically via curl rather than manually entering the cookie info for the registry test script
Add the code to check if the existing highest uidNumber really exists.
In the getNextUidNumber method, a mechanism to look up the highest existing udiNumber was added.
Fix XML validation errors in the metacatui confirmData template for the registry. Add a test script that submits multiple datasets to the registry.
Create a lock file for the registry if one doesn't exist
Allow the registry form to specify a docid scope
Only lock the local docid file when creating a new docid, not when inserting, for faster upload times. Remove extra debug messages from testing.
Fix bug in the online registry where data files were not using the new docid creation process
Lock a local file while docids are being created so multiple docs can be uploaded at once
remove CN.systemMetadataChanged in favor of the CN.updateSystemMetadata method. Otherwise there's no good way to know where to fetch the auth copy from since the SM change might be to switch the authMN!
add support for v2 DataONE API.
remove old EML jar -- datamanager.jar has the EMLParser now and is pulled in with maven.
use css changes from EML project to render a PDF that fits on a printed page during export. Note that this also changes the default skin slightly (for the better, we think). https://projects.ecoinformatics.org/ecoinfo/issues/6053
remove configxml.jar as the ConfigXML class is now included in the utilities library.
handle login/logout when testing using metacat client (recent refactoring to use more standard http client code)
handle multiple subprocessors (RDF and ORE) before the object gets indexed by making sure to merge the solr doc map before submitting to the index.
use maven for datamanager dependency now that EML project has been [partially] maven-ized.
switch to use utilities from Maven repo instead of pulling from SVN and building locally. Added trunk/SNAPSHOT build to dev-testing Hudson (DataONE) for users who do not actively develop the utilities library. Motivated by the HTML->PDF utility, seemed like as good a time as any to convert Metacat's dependency on the project.
remove dependency on HttpMessage that was in the utilities project but is now removed in favor of newer (standard) http client library code.
add more sections about extending the annotation model, serializations, and permissions.
Include PDF version of the metadata in the package download. https://projects.ecoinformatics.org/ecoinfo/issues/6053
use default "metacat" context name for metacat-index testing.
Added more description for the model. cleaned up a few formatting issues.
draft of semantic model documentation
Change geohash type to "string" (was "text") so that no Solr analysis or tokenizing happens during indexing.
take advantage of the ezidclient for multi-threaded/asynchronous DOI registration. This will be most useful for doing large batch updates and not so much for the one-at-a-time publish actions but works in either context. https://projects.ecoinformatics.org/ecoinfo/issues/6440
use a member instance of ezid service that only logs in every 24 hours (or other time TBD) instead of every time there is an interaction with the service. Saves us many calls when doing batch updates to ezid but keeps us from trying to use expired sessions. Motivated by https://projects.ecoinformatics.org/ecoinfo/issues/6440
Fixed an error in the ORE model diagrams
prevent js scriptlets from running when we return error messages to the client by escaping any potentially harmful xml blocks. https://projects.ecoinformatics.org/ecoinfo/issues/6224
allow updates to all doi: prefixes - realized we are already restricting to specific replica servers when updating these. worst case is we try to update a registration for which we are not the owner. https://projects.ecoinformatics.org/ecoinfo/issues/6440
show the SM and ORE generation buttons even if they have not registered/configured dataone. many potential MNs want to see their generated SM before registering (and we want them to too!).
restrict DOI updates to DOIs that match our server shoulder -- may consider opening this up to any "doi:" prefix if this is too restrictive. https://projects.ecoinformatics.org/ecoinfo/issues/6440
use separate surName and givenNames to lookup ORCIDs.
Changed errors in the ORE model expansion metacat documentation diagrams
Describe the primary data resource in the derived resource RDF
Create a derived data and metadata in the OrePackageTest
all full-text queries for ORCID, but it isn't that great because we might have a"PISCO" creator that shows us in may different orcid profiles...false matches.
correct glaring erros -- still needs to be honed, but at least it runs without NPE and Jena/foresite errors
stub for testing ORE augmentation - this generates an ORE, adds a "wasDerivedFrom" triple and saves to Metacat MN for indexing. https://projects.ecoinformatics.org/ecoinfo/issues/6548
include ORE formatId as handled by the RDF subprocessor and index prov:wasDerivedFrom field where it exists in the RDF model. https://projects.ecoinformatics.org/ecoinfo/issues/6548
expand the sparql queries to include dcterms:identifier
Text changes to ORE docs
Add documentation for the ORE model expansion for derived data
use HttpClient to query orcid so I can easily set headers and such -- getting 503s from their production server when I test on dev.nceas...odd
adjust tests for production service -- more "real" information shows additional return values from the query.
switch to the production ORCID server for looking up orcid matches for our creators.add test to summarize how many creator matches we can actually find. https://projects.ecoinformatics.org/ecoinfo/issues/6423
use a non-public rightsHolder for both EML and Annotation test documents now that the RDF subprocessor checks each annotation to see that it came from a user that as write permission for the object being annotated.
test for update using the updated EML file, not the original. Also add the SM to the shared map so that the indexing process can consult SM.accessPolicy when indexing annotations that assert things about those test documents.
ignore the metacat/solr comparator tests - they are one-offs.
change the hazelcast group name to be the default "metacat" instance so that the metacat-index tests pass without additional local configuration, at least when running a default metacat deployment.
do not set archived=false for all CN.create calls. The CN will use create() even harvesting content that is new to it and needs to handle already-archived content. https://projects.ecoinformatics.org/ecoinfo/issues/6475
cache the imported models to avoid timeouts from remote hosts (or being locked out for too many requests in a given time period).
process all the returned annotation suggestions until we find one that is appropriately located in the subclass hierarchy for the given superclass.
use in-memory TDB dataset for querying annotations for indexing -- this comes with the same reasoning capabilities as the directory-based one, but has the benefit of not filling the directory with triples that will not be used again. prepping for d1 AHM
pass around the object file path rather than the data stream so that multiple subprocessors can index the same object and not consume the stream before it gets to the next one. In preparation for extending the assertions stored in OREs. https://projects.ecoinformatics.org/ecoinfo/issues/6548
when indexing annotations directly, just use an in-memory triple store rather than TDB since we remove each graph as it is processed (and my TDB instance would get into the multi-GB range with a few runs, even if I removed the old models)
redirect "short form" metacat read URIs to the the new Metacat UI using the configured UI context. This translates the docid -> pid to use the correct identifier for the correct service. https://projects.ecoinformatics.org/ecoinfo/issues/6546
simplify lookup for classes and orcid. remove the "random" annotation code branches -- just too confusing to look at those bogus classes especially now that we have "real" generated annotations.
add 'test' for indexing annotations without actually storing the RDF of the generated annotation.
only allow multiple values for multi-valued fields....
Add admin service to update DOI registrations by specifying a list of formatIds or DOIs, or update all.
use new method to override the CN URL when constructing a CNode instance. see https://redmine.dataone.org/issues/5142
use newer httpclient library so that Jena's dependency is met - this goes all the way back to d1_common/libclient needing to pull in the newer library.
first pass at direct EML->semantic index method. Still produces an RDF model, but does not persist it in Metacat, only in the triplestore. Allows us to re-run without adding stale RDF to the MN store.
allow multivalued fields to be indexed using the "fields" pass through.
Remove the attribute disable from the update button if it has been submitted.
The image which has the default values.
Add a new screen shot which contains the cn url.
Localized the file which doesn't have the bean for dataUrl.
Add the cn url.
Add the text field for the cn url.
Store the cn url in the backup.
switch to use FIleUpload instead of O'Reilly COS library for handling chunked file uploads. https://projects.ecoinformatics.org/ecoinfo/issues/6517
forgot to check in the actual class: first pass at allowing admins to update DOI registration. This only acts on EML objects at the moment and is meant to illustrate one mechanism for updating the DOIs. https://projects.ecoinformatics.org/ecoinfo/issues/6530
first pass at allowing admins to update DOI registration. This only acts on EML objects at the moment and is meant to illustrate one mechanism for updating the DOIs. https://projects.ecoinformatics.org/ecoinfo/issues/6530
correct the ORE lookup query syntax and add junit assertion to check that it continues to function as expected. https://projects.ecoinformatics.org/ecoinfo/issues/6529
index the ORE after we submit the metadata for indexing. https://projects.ecoinformatics.org/ecoinfo/issues/6520
include BioPortal lookup for Entity matches using the data table description. TODO: only associate measurements to the entity observation if they apply.