Project

General

Profile

Statistics
| Revision:

# Date Author Comment
8973 11/18/2014 02:39 PM ben leinfelder

use sparql field and triple store from cn-index_processor (refactor). include annotatorSubprocessor for testing in metacat-index

8893 10/10/2014 03:04 PM Jing Tao

when we remove a slor index of a resource map, we don't need to know the content of the resource map. Instead, we will search the solr index to get information.

8864 09/16/2014 05:07 PM Jing Tao

Add the code to handle to remove the resource map index.

8862 09/16/2014 04:48 PM Jing Tao

Add codes to handle remove a source map solr index.

8857 09/15/2014 02:32 PM Lauren Walker

Create a valid URI by using all lowercase letters when creating a name for the triple model in the Rdf Xml Subprocessor. See bug: https://projects.ecoinformatics.org/ecoinfo/issues/6595

8840 08/19/2014 11:38 AM Lauren Walker

When indexing annotations from RDFs, use the doc id to access the system metadata, not the model name since they are not always the same.

8839 08/18/2014 03:25 PM Lauren Walker

Add PROV relationships to the Solr schema. Populate the fields using the RdfXmlSubprocessor

8826 08/06/2014 01:09 PM ben leinfelder

update to use v2 types for indexing

8805 05/30/2014 05:06 PM ben leinfelder

handle multiple subprocessors (RDF and ORE) before the object gets indexed by making sure to merge the solr doc map before submitting to the index.

8774 05/13/2014 03:36 PM ben leinfelder

use a non-public rightsHolder for both EML and Annotation test documents now that the RDF subprocessor checks each annotation to see that it came from a user that as write permission for the object being annotated.

8766 05/07/2014 11:16 PM ben leinfelder

pass around the object file path rather than the data stream so that multiple subprocessors can index the same object and not consume the stream before it gets to the next one. In preparation for extending the assertions stored in OREs. https://projects.ecoinformatics.org/ecoinfo/issues/6548

8761 04/30/2014 03:46 PM ben leinfelder

only allow multiple values for multi-valued fields....

8756 04/29/2014 04:52 PM ben leinfelder

allow multivalued fields to be indexed using the "fields" pass through.

8722 04/02/2014 03:35 PM ben leinfelder

handle null Boolean in SM.archived field

8701 03/14/2014 10:55 AM ben leinfelder

switch to index standard since it is more likely we will be able to determine this from our existing EML attribute information. https://projects.ecoinformatics.org/ecoinfo/issues/6253

8692 03/03/2014 03:56 PM ben leinfelder

switch to the OpenAnnotation (OA) model for annotating datapackages with measurements/characteristics (semtools)

8580 02/07/2014 02:20 PM ben leinfelder

include ID field as a minimum for indexing additional fields.

8554 02/03/2014 02:53 PM ben leinfelder

correctly include stacktrace for error debugging.

8552 02/03/2014 02:51 PM ben leinfelder

return null if there is no existing SolrDoc for the given pid.

8503 01/15/2014 09:55 AM ben leinfelder

check for existing index document before trying to use existing fields.

8498 01/14/2014 02:57 PM ben leinfelder

allow indexing of RDF documents - provide a sparql query that will return values for the field name. Using measurement_sm initially (a dynamic multivalued solr field). https://projects.ecoinformatics.org/ecoinfo/issues/6253

8496 01/14/2014 02:14 PM ben leinfelder

check for existing documents - don't assume it exists.

8464 01/07/2014 01:56 PM ben leinfelder

Unify solr indexing with an IndexTask that is added to the queue -- allows us to send more than just the systemMetadata to the indexer. Initially this is for READ event counts for each document. https://projects.ecoinformatics.org/ecoinfo/issues/6346

8352 10/30/2013 10:19 AM Jing Tao

Rename the IndexGenerator to IndexGeneratorTimerTask.

8345 10/28/2013 08:21 PM Jing Tao

Fixed a bug that when a data file was archived, the solr index for the metadata object still kept the "documents" element.

8344 10/28/2013 05:49 PM Jing Tao

made the delete method synchronized.

8343 10/28/2013 04:54 PM Jing Tao

If an object was archived, the solr index will be removed for it.

8310 10/11/2013 11:07 AM Jing Tao

Use the setting from the metacat-common component.

8296 10/09/2013 12:10 PM Jing Tao

Use the d1_cn_index_processor 1.2.0 version.

8293 10/08/2013 05:51 PM Jing Tao

combine the index code for failed ids and other ids.

8292 10/08/2013 04:49 PM Jing Tao

Clean up the code.

8291 10/08/2013 02:22 PM Jing Tao

The IndexGenerator will index the obsoleted data objects as well.

8288 10/08/2013 11:33 AM Jing Tao

Remove the obsoletes chain from the update method in the SolrIndex class.

8287 10/07/2013 05:56 PM Jing Tao

When an object is archvied, the solr index will not be removed.

8283 10/04/2013 09:26 AM ben leinfelder

merge from 2.2 branch: remove the index queue item when it is being processed. https://projects.ecoinformatics.org/ecoinfo/issues/6117

8187 09/12/2013 10:45 AM ben leinfelder

remove any index event errors if the pid has successfully been reindexed. https://projects.ecoinformatics.org/ecoinfo/issues/6089

8161 08/23/2013 11:39 AM Jing Tao

Change the parameters order of the constructor. We maybe reuse some code from d1_cn_processor.

8138 08/09/2013 05:07 PM Jing Tao

Modified the documentation.

8136 08/09/2013 04:52 PM Jing Tao

Modified the documentation.

8134 08/09/2013 04:43 PM Jing Tao

Use the ResourceMapException when a component of a resource map isn't found in the solr index.

8133 08/09/2013 04:42 PM Jing Tao

Add a ResourceMapException.

8128 08/08/2013 03:08 PM Jing Tao

Use the class path configuration of spring to replace the file configuration. We can reuse the application context files in the d1_cn_index_processor jar.

8125 08/08/2013 01:43 PM Jing Tao

Add a constructor.

8124 08/08/2013 11:03 AM Jing Tao

Remove the constructor.

8084 07/30/2013 10:26 AM Jing Tao

Remove a logFile method.

8069 07/22/2013 03:48 PM ben leinfelder

use the v1.1.x branch ResourceMap class for metacat-index

8035 07/19/2013 09:08 AM Jing Tao

The exceptions will be caught during the looping of deleting the solr index.

8027 07/18/2013 09:06 AM Jing Tao

Remove the code to write some debug information into a temporary file.

8023 07/17/2013 03:48 PM Jing Tao

Use the ResourceMapFactory rather than the ResourceMap constructor to build a resource map.

8017 07/17/2013 11:33 AM Jing Tao

Write the ids from metacat into a temporary file.

7931 07/12/2013 09:06 AM Jing Tao

Move a file to the temp dir.

7924 07/11/2013 04:28 PM Jing Tao

Add a method to write ids which will be indexed into a file.

7877 07/10/2013 04:32 PM Jing Tao

Besides the getArchvied() method, the getObsoletedBy method was added to determine if the object is archvied or not.

7876 07/10/2013 04:14 PM Jing Tao

Add code to handle deleted ids.

7866 07/09/2013 02:37 PM Jing Tao

Use schedule method to start the index.

7863 07/09/2013 10:16 AM Jing Tao

Add the code to write the error message to the log in the itemRemvoed method.

7858 07/08/2013 03:09 PM Jing Tao

In determining the time arrange, the equality was removed.

7857 07/08/2013 03:05 PM Jing Tao

Add code to handle failed ids.

7856 07/08/2013 02:00 PM Jing Tao

Remove the EventLog write.

7854 07/08/2013 12:09 PM Jing Tao

Add the EventLog code.

7852 07/05/2013 05:14 PM Jing Tao

It will throw an exception if the subprocessor can't handle the document.

7851 07/05/2013 04:59 PM Jing Tao

Check if the all components of a resource map have been processed before processing the resource map.

7848 07/03/2013 04:34 PM Jing Tao

Fixed a bug that the event log can't save the real lastest process date.

7847 07/03/2013 03:56 PM Jing Tao

Change the date format.
Remove the replication part of log4j.

7846 07/03/2013 10:28 AM Jing Tao

Use a new date format.

7838 07/02/2013 03:21 PM Jing Tao

Add the code that only the ids with the correct system metadata modification time will be added to the index queue.

7836 07/01/2013 01:59 PM Jing Tao

Add code to get and set the last process date.

7828 06/25/2013 01:23 PM ben leinfelder

move IndexEvent into metacat-common. Perparation for Metacat responding to events and writing them to a persistent store. https://projects.ecoinformatics.org/ecoinfo/issues/5944

7815 06/21/2013 09:09 AM ben leinfelder

refactor IndexEventLog a bit to simplify type/action information. prep for serializing IndexEvent objects to Metacat. https://projects.ecoinformatics.org/ecoinfo/issues/5944

7814 06/20/2013 05:28 PM ben leinfelder

remove serial number from indexeventlog - it is not used elsewhere in the api. https://projects.ecoinformatics.org/ecoinfo/issues/5944

7813 06/20/2013 05:21 PM ben leinfelder

correct spelling for index.eventlog.classname property

7812 06/20/2013 04:49 PM ben leinfelder

use an independent ISet<SystemMetadata> structure to communicate objects that should be indexed by metacat-index. https://projects.ecoinformatics.org/ecoinfo/issues/5943

7811 06/20/2013 12:23 PM ben leinfelder

consolidate SystemMetadata map retrieval in preparation for using a different structure for objects to index.

7810 06/20/2013 09:24 AM ben leinfelder

adding ability to remove event from the [error] queue.

7807 06/17/2013 05:07 PM Jing Tao

Add code to implment set and get the last processed date.

7806 06/17/2013 04:28 PM Jing Tao

It will make the index only for those objects which were modified after the marked time.

7805 06/17/2013 01:50 PM Jing Tao

Add set and get the lastprocessedDate in the IndexEventLog.
Remove the code to write the successful event.

7802 06/12/2013 02:36 PM Jing Tao

Log the timed index jobs.

7801 06/12/2013 02:08 PM Jing Tao

Add the code to log the failed events.

7800 06/11/2013 06:24 PM Jing Tao

Add a temporary file log for debugging.

7797 06/07/2013 04:38 PM Jing Tao

Add a serial number for the event.
Add method to set events to be archived.

7796 06/07/2013 11:22 AM Jing Tao

Add a new class variable - isArchived for class IndexEvent.

7795 06/06/2013 04:13 PM Jing Tao

Update the documentation about those classes.

7794 06/06/2013 03:17 PM Jing Tao

Add a event and eventlog for the index.

7793 06/05/2013 03:03 PM Jing Tao

Use the identifier set to get the list of ids in the member node.

7792 06/05/2013 02:20 PM Jing Tao

The returned ISet should be Identifier.

7790 06/05/2013 01:50 PM Jing Tao

Add method to get identifier set.

7788 06/04/2013 02:47 PM Jing Tao

Set up a Timer to run the regenerating solr index task periodically.

7786 05/31/2013 03:29 PM Jing Tao

Add code to handle delete data package information when delete a pid in the solr index.

7785 05/31/2013 02:05 PM Jing Tao

Add two static methods to get the SystemMetadata and data object InputStream for the specified id.

7783 05/28/2013 05:18 PM Jing Tao

Add code to check if the metacat.properties is available.

7782 05/28/2013 01:21 PM Jing Tao

If solr is not enabled, it would not be running.

7776 05/23/2013 11:57 PM Jing Tao

Use another thread in the Servlet init method to wait hazelcast.

7774 05/23/2013 09:59 PM Jing Tao

Put the waiting mechanism for the hazelcast at the first place.

7771 05/23/2013 06:28 PM Jing Tao

Temporarily remove the code to disable solr engine if it isn't listed in the property file.

7769 05/23/2013 04:27 PM Jing Tao

If the solr engine is disabled, the metacat index will do nothing.

7754 05/22/2013 04:18 PM Jing Tao

set the dataone.hazelcast.location.clientconfig property to be the local config file. It will be used in the d1_cn_index_processor

7748 05/22/2013 11:19 AM Jing Tao

Read the waiting time and maximum attempts from the metacat.properties file.

7740 05/21/2013 11:33 PM Jing Tao

Index the none-resourcemap data objects first, then the resourcemap objects.

7734 05/21/2013 02:23 PM Jing Tao

Try to get the metacat ids first.