Project

General

Profile

Activity

From 05/27/2013 to 06/25/2013

06/25/2013

04:42 PM Revision 7830 (metacat): upgrade to Metacat 2.1.0 on the trunk. This includes a new index_event table for storing indexing events that need to be reprocessed. https://projects.ecoinformatics.org/ecoinfo/issues/5944
ben leinfelder
02:51 PM Task #5915 (Closed): Use MN.query(solr) in NCEAS skin
ben leinfelder
02:50 PM Task #5918 (Closed): Separate unit tests from integration tests
ben leinfelder
02:43 PM Bug #6007 (Resolved): Registry uses filename from upload form to save in temporary directory
I deployed this in KNB today (June 25th, 2013). This was not part of an official release but is in the code for 2.1.x... ben leinfelder
01:23 PM Revision 7829 (metacat): stub for storing IndexEvent objects in Metacat (from metacat-index processing). https://projects.ecoinformatics.org/ecoinfo/issues/5944
ben leinfelder
01:23 PM Revision 7828 (metacat): move IndexEvent into metacat-common. Perparation for Metacat responding to events and writing them to a persistent store. https://projects.ecoinformatics.org/ecoinfo/issues/5944
ben leinfelder
01:17 PM Revision 7827 (metacat): do not force a get() during refresh (causing EML-defined data access rules to be lost when inserting EML docs about data files). note that this reverses a change that was meant to trigger indexing, but now we are using a new queue to share index events with metacat-index and so should not be necessary.
ben leinfelder
09:10 AM Bug #6009 (Closed): Metacat uses "metacat.output" temp files for all read() operations
This is in trunk now. ben leinfelder

06/24/2013

12:07 PM Bug #6009 (In Progress): Metacat uses "metacat.output" temp files for all read() operations
now reading directly from the filesystem, or in the case of EML with inline-data, the partial file contents. ben leinfelder
12:05 PM Revision 7824 (metacat): do not use tmp file to return an inputstream on read() operations - just read from the file we already have. https://projects.ecoinformatics.org/ecoinfo/issues/6009
ben leinfelder

06/21/2013

09:47 PM Bug #6008 (Closed): Metacat servlet API upload() method uses user-supplied filename for temp upload
Now using File.createTempFile() and deleting the data files when done. ben leinfelder
05:32 PM Bug #6008 (Closed): Metacat servlet API upload() method uses user-supplied filename for temp upload
This can be problematic if users simultaneously attempt to upload a data file with the same name. ben leinfelder
09:44 PM Revision 7823 (metacat): use standard File.createTempFile() method for uploaded data files and delete them when we are done with them. https://projects.ecoinformatics.org/ecoinfo/issues/6008
ben leinfelder
05:35 PM Bug #6009 (Closed): Metacat uses "metacat.output" temp files for all read() operations
The KNB has 1.3 GB of these that have accumulated since ?.
There should be two scenarios:
-stream the contents fr...
ben leinfelder
05:02 PM Bug #6007: Registry uses filename from upload form to save in temporary directory
I deployed the changes on dev2.nceas for testing. Feel pretty good about them - using File::Temp to manager unique up... ben leinfelder
12:22 PM Bug #6007 (Resolved): Registry uses filename from upload form to save in temporary directory
This can cause collisions when a room of 22 people all submit a datapackage with the same data file name....
The r...
ben leinfelder
04:15 PM Revision 7822 (metacat): correctly delete data file when we are done with it. https://projects.ecoinformatics.org/ecoinfo/issues/6007
ben leinfelder
04:06 PM Revision 7821 (metacat): include filename in the filepart part. https://projects.ecoinformatics.org/ecoinfo/issues/6007
ben leinfelder
03:18 PM Revision 7820 (metacat): send the original filename in the upload() method since that is supported by the Metacat API. https://projects.ecoinformatics.org/ecoinfo/issues/6007
ben leinfelder
03:10 PM Revision 7819 (metacat): remove the unique string when generating data file metadata. https://projects.ecoinformatics.org/ecoinfo/issues/6007
ben leinfelder
02:48 PM Revision 7818 (metacat): debugging. https://projects.ecoinformatics.org/ecoinfo/issues/6007
ben leinfelder
01:44 PM Revision 7817 (metacat): use File::Temp to write data files in registry. https://projects.ecoinformatics.org/ecoinfo/issues/6007
ben leinfelder
11:11 AM Revision 7816 (metacat): correct regex for whitespace in D1 identifier.
ben leinfelder
09:09 AM Revision 7815 (metacat): refactor IndexEventLog a bit to simplify type/action information. prep for serializing IndexEvent objects to Metacat. https://projects.ecoinformatics.org/ecoinfo/issues/5944
ben leinfelder

06/20/2013

05:28 PM Revision 7814 (metacat): remove serial number from indexeventlog - it is not used elsewhere in the api. https://projects.ecoinformatics.org/ecoinfo/issues/5944
ben leinfelder
05:28 PM Bug #6006 (Closed): Metacat admin screen requires metacat-index directories
Now skipping the template creation if metacat-index war is not deployed along with metacat. ben leinfelder
05:21 PM Revision 7813 (metacat): correct spelling for index.eventlog.classname property
ben leinfelder
04:59 PM Task #5943 (Resolved): Eliminate the unnecessary Systemmetadata event between the Metacat and Metacat-index modules.
Switched over to the new structure for communicating index events. Seems to be working as expected. ben leinfelder
04:49 PM Revision 7812 (metacat): use an independent ISet<SystemMetadata> structure to communicate objects that should be indexed by metacat-index. https://projects.ecoinformatics.org/ecoinfo/issues/5943
ben leinfelder
12:23 PM Revision 7811 (metacat): consolidate SystemMetadata map retrieval in preparation for using a different structure for objects to index.
ben leinfelder
10:22 AM Task #5944: Metacat-index need an IndexEvent and IndexEventLog mechanism
Added delete() method to remove entries that we do not need to reprocess again (because they were successful, finally... ben leinfelder
09:24 AM Revision 7810 (metacat): adding ability to remove event from the [error] queue.
ben leinfelder

06/18/2013

02:59 PM Revision 7809 (metacat): do not create solr-home if there is no template to compy into that directory (need to be able to create it later if/when someone decides to use and deploy metacat-index). https://projects.ecoinformatics.org/ecoinfo/issues/6006
ben leinfelder
02:54 PM Task #5943: Eliminate the unnecessary Systemmetadata event between the Metacat and Metacat-index modules.
Proposed solution is to use a different/independent structure for SystemMetadata events that should trigger a re-inde... ben leinfelder
01:54 PM Revision 7808 (metacat): do not attempt to copy solr-home template from metacat-index webapp if it does not exist. This would be in cases where metacat-index is not deployed. https://projects.ecoinformatics.org/ecoinfo/issues/6006
ben leinfelder
12:35 PM Bug #6006 (Closed): Metacat admin screen requires metacat-index directories
When configuring Metacat there is a dependency on metacat-index. This should be gracefully skipped if that webapp is ... ben leinfelder
10:19 AM Task #6005 (Closed): Incomplete solr index for the data objects if the resourcemap was indexed first
Currently the resource map should be indexed after the data objects in the resource map were indexed. If the resource... Jing Tao

06/17/2013

05:07 PM Revision 7807 (metacat): Add code to implment set and get the last processed date.
Jing Tao
04:28 PM Revision 7806 (metacat): It will make the index only for those objects which were modified after the marked time.
Jing Tao
01:50 PM Revision 7805 (metacat): Add set and get the lastprocessedDate in the IndexEventLog.
Remove the code to write the successful event. Jing Tao
10:31 AM Task #6004: Figure out why there are only 422 documents indexed in mn-demo-4.test.dataone.org
Attached a log file for the indexing process. Jing Tao
10:30 AM Task #6004 (Resolved): Figure out why there are only 422 documents indexed in mn-demo-4.test.dataone.org
There are more 4000 data objects in the mn-demo-4. But we only gets 422 documents indexed. Jing Tao

06/13/2013

05:39 PM Bug #6003 (Closed): The "war" target in the metacat will build the metacat-index.war
We added the target which build metacat-index.war as the dependency of the "war" target. Jing Tao
05:37 PM Bug #6003 (Closed): The "war" target in the metacat will build the metacat-index.war
The "war" target in the metacat doesn't build the metacat-index.war in the dist directory. It only builds knb.war and... Jing Tao
05:35 PM Task #5930: Add a configuration property to turn on/off the indexing processes (SOLR indexing)
If we removed the "solr" from the engine list, metacat-index will not build index anymore. And the solr query will be... Jing Tao
05:20 PM Revision 7804 (metacat): Add the dataone repository.
Jing Tao
04:41 PM Revision 7803 (metacat): The "war" target will build the metacat-index.war as well.
Jing Tao

06/12/2013

02:36 PM Revision 7802 (metacat): Log the timed index jobs.
Jing Tao
02:08 PM Revision 7801 (metacat): Add the code to log the failed events.
Jing Tao

06/11/2013

06:24 PM Revision 7800 (metacat): Add a temporary file log for debugging.
Jing Tao
06:23 PM Revision 7799 (metacat): Use commons-io 2.4
Jing Tao
03:47 PM Revision 7798 (metacat): Add a new property for the index event log class name.
Jing Tao

06/07/2013

04:38 PM Revision 7797 (metacat): Add a serial number for the event.
Add method to set events to be archived. Jing Tao
11:56 AM Feature #5936: Include certificate delegation inside Metacat
There's now the package "edu.ucsb.nceas.metacat.portal" that does the authentication delegation. You start a request ... ben leinfelder
11:22 AM Revision 7796 (metacat): Add a new class variable - isArchived for class IndexEvent.
Jing Tao

06/06/2013

04:13 PM Revision 7795 (metacat): Update the documentation about those classes.
Jing Tao
03:48 PM Bug #5997 (Closed): Restrict KNB trusted CAs
Commented out the line that includes all CAs in /etc/ssl/certs and reloaded Apache. Now we are down to the short list... ben leinfelder
03:17 PM Revision 7794 (metacat): Add a event and eventlog for the index.
Jing Tao

06/05/2013

03:03 PM Revision 7793 (metacat): Use the identifier set to get the list of ids in the member node.
Jing Tao
02:20 PM Revision 7792 (metacat): The returned ISet should be Identifier.
Jing Tao
01:50 PM Revision 7791 (metacat): Add code to test getIdentifierSet method.
Jing Tao
01:50 PM Revision 7790 (metacat): Add method to get identifier set.
Jing Tao
11:02 AM Bug #5997: Restrict KNB trusted CAs
We should be able to simply use the DataONE chain file that we have installed already:... ben leinfelder
11:00 AM Bug #5997 (Closed): Restrict KNB trusted CAs
Instead of trusting all commercial CAs, the KNB Member Node should only trust the DataONE and CILogon certificate aut... ben leinfelder

06/04/2013

02:49 PM Revision 7789 (metacat): Add a new property to specify the interval of a Timer to run the thread generating solr index.
Jing Tao
02:47 PM Revision 7788 (metacat): Set up a Timer to run the regenerating solr index task periodically.
Jing Tao

06/03/2013

05:36 PM Task #5922 (Closed): Modify the solr index configuration for the resource map
We rewrote the ResourceMapSubprocessor class which now uses the SolrServer. Jing Tao
05:22 PM Task #5937 (Closed): The solr indexes of the data file contains obsoleted ids for the resourcemap and documentBy element after updating a data package
Add code to modify the "documents", "resourceMap" and "documentedBy" element in the solr doc when archive happens. Jing Tao
04:35 PM Revision 7787 (metacat): Use the ";" as the seperator to replace "," in the resource name spaces.
Jing Tao

05/31/2013

03:29 PM Revision 7786 (metacat): Add code to handle delete data package information when delete a pid in the solr index.
Jing Tao
02:05 PM Revision 7785 (metacat): Add two static methods to get the SystemMetadata and data object InputStream for the specified id.
Jing Tao

05/28/2013

05:19 PM Revision 7784 (metacat): Change the code since the ApplicationController's constructor was changed.
Jing Tao
05:18 PM Revision 7783 (metacat): Add code to check if the metacat.properties is available.
Jing Tao
01:21 PM Revision 7782 (metacat): If solr is not enabled, it would not be running.
Jing Tao
01:20 PM Revision 7781 (metacat): Solr will be enabled if it is in the db.enabledEngines.
Jing Tao
01:18 PM Revision 7780 (metacat): Use ";" to separate db.enabledEngines.
Jing Tao
 

Also available in: Atom