https://projects.ecoinformatics.org/ecoinfo/https://projects.ecoinformatics.org/ecoinfo/ecoinfo/favicon.ico?14691340362010-04-27T10:24:26ZEcoinformatics RedmineMetacat - Bug #3835: design and implement OAI-PMH compliant harvest subsystemhttps://projects.ecoinformatics.org/ecoinfo/issues/3835?journal_id=129782010-04-27T10:24:26ZHannu Saarenmaahannu.saarenmaa@helsinki.fi
<ul></ul><p>I would be curious to hear what is the status of these developments now? We are very much looking for them.</p>
<p>My view on the issues is to offer a simple solution first and then see if more functionality is really needed. So, use both Metacat and OAI-PMH protocols in parallel. The latter would have a just a public read-only interface, while actions that require authentication or updates would have to be performed via Metacat protocol. For point 3) on change detection, only serve the latest and not intermediate versions.</p> Metacat - Bug #3835: design and implement OAI-PMH compliant harvest subsystemhttps://projects.ecoinformatics.org/ecoinfo/issues/3835?journal_id=129792010-05-13T20:33:42ZMatt Jonesjones@nceas.ucsb.edu
<ul></ul><p>Hi Hannu --<br />Regarding Comment <a class="issue tracker-1 status-3 priority-5 priority-highest closed" title="Bug: need more extensive element documentation (Resolved)" href="https://projects.ecoinformatics.org/ecoinfo/issues/1">#1</a>, I've inquired over email about the status of this feature with Duane Costa, who developed it. As far as I know the OAI-PMH is functional in Metacat -- hopefully Duane will fill in the details. I noticed that it is not documented in the Metacat Administrator's Guide, which we will add to our TODO list for the 1.10 release.</p> Metacat - Bug #3835: design and implement OAI-PMH compliant harvest subsystemhttps://projects.ecoinformatics.org/ecoinfo/issues/3835?journal_id=129802010-05-14T15:03:02ZDuane Costadcosta@lternet.edu
<ul></ul><p>Hi Matt,</p>
<p>Support for OAI-PMH is included in the Metacat distribution as of version 1.9.2. Configuring Metacat as an OAI-PMH data provider requires that the Metacat administrator support the service by configuring a set of OAI-PMH properties and activating the 'dataProvider' servlet in the web deployment descriptor file. Support for OAI-PMH harvesting from a remote data repository into Metacat is also provided.</p>
<p>Documentation is provided in file 'docs/dev/oaipmh/MetacatOaipmh.pdf'. In a metacat-dev email dated 4/16/2009 ('Re: [metacat-dev] Proposed Dryad project integration with Metacat'), we had agreed to place design and planning documents in directory 'metacat/docs/dev/oaipmh', but we had not yet addressed the issue of the Metacat Administrator's Guide. I think the thinking at the time was that this was not yet a mature product, but I agree that it's now time to add user documentation to the Administrator's Guide and to officially support this feature in the next Metacat release, especially given the increased level of demand.</p>
<p>The following links demonstrate its use on a Metacat 1.9.2 test instance for which the OAI-PMH service has been configured and deployed:</p>
<p><a class="external" href="http://scoria.lternet.edu:8080/knb/dataProvider?verb=Identify">http://scoria.lternet.edu:8080/knb/dataProvider?verb=Identify</a></p>
<p><a class="external" href="http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListMetadataFormats">http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListMetadataFormats</a></p>
<p><a class="external" href="http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListIdentifiers&metadataPrefix=oai_dc&from=2001-01-01&until=2010-01-01">http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListIdentifiers&metadataPrefix=oai_dc&from=2001-01-01&until=2010-01-01</a><br /><a class="external" href="http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListIdentifiers&metadataPrefix=eml-2.0.0&from=2001-01-01&until=2010-01-01">http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListIdentifiers&metadataPrefix=eml-2.0.0&from=2001-01-01&until=2010-01-01</a><br /><a class="external" href="http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListIdentifiers&metadataPrefix=eml-2.0.1&from=2001-01-01&until=2010-01-01">http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListIdentifiers&metadataPrefix=eml-2.0.1&from=2001-01-01&until=2010-01-01</a><br /><a class="external" href="http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListIdentifiers&metadataPrefix=eml-2.1.0&from=2001-01-01&until=2010-01-01">http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListIdentifiers&metadataPrefix=eml-2.1.0&from=2001-01-01&until=2010-01-01</a></p>
<p><a class="external" href="http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListRecords&metadataPrefix=oai_dc">http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListRecords&metadataPrefix=oai_dc</a><br /><a class="external" href="http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListRecords&metadataPrefix=eml-2.0.0">http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListRecords&metadataPrefix=eml-2.0.0</a><br /><a class="external" href="http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListRecords&metadataPrefix=eml-2.0.1">http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListRecords&metadataPrefix=eml-2.0.1</a><br /><a class="external" href="http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListRecords&metadataPrefix=eml-2.1.0">http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListRecords&metadataPrefix=eml-2.1.0</a></p>
<p><a class="external" href="http://scoria.lternet.edu:8080/knb/dataProvider?verb=GetRecord&metadataPrefix=oai_dc&identifier=urn:lsid:knb.ecoinformatics.org:knb-lter-gce:26">http://scoria.lternet.edu:8080/knb/dataProvider?verb=GetRecord&metadataPrefix=oai_dc&identifier=urn:lsid:knb.ecoinformatics.org:knb-lter-gce:26</a><br /><a class="external" href="http://scoria.lternet.edu:8080/knb/dataProvider?verb=GetRecord&metadataPrefix=eml-2.0.0&identifier=urn:lsid:knb.ecoinformatics.org:knb-lter-and:4056">http://scoria.lternet.edu:8080/knb/dataProvider?verb=GetRecord&metadataPrefix=eml-2.0.0&identifier=urn:lsid:knb.ecoinformatics.org:knb-lter-and:4056</a><br /><a class="external" href="http://scoria.lternet.edu:8080/knb/dataProvider?verb=GetRecord&metadataPrefix=eml-2.0.1&identifier=urn:lsid:knb.ecoinformatics.org:knb-lter-gce:26">http://scoria.lternet.edu:8080/knb/dataProvider?verb=GetRecord&metadataPrefix=eml-2.0.1&identifier=urn:lsid:knb.ecoinformatics.org:knb-lter-gce:26</a><br /><a class="external" href="http://scoria.lternet.edu:8080/knb/dataProvider?verb=GetRecord&metadataPrefix=eml-2.1.0&identifier=urn:lsid:knb.ecoinformatics.org:knb-lter-mcr:7">http://scoria.lternet.edu:8080/knb/dataProvider?verb=GetRecord&metadataPrefix=eml-2.1.0&identifier=urn:lsid:knb.ecoinformatics.org:knb-lter-mcr:7</a></p>
<p><a class="external" href="http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListSets">http://scoria.lternet.edu:8080/knb/dataProvider?verb=ListSets</a></p>
<p>The base URL for OAI-PMH harvesting from the data provider in the above examples is 'http://scoria.lternet.edu:8080/knb/dataProvider'. Note that 'eml-2.0.0', 'eml-2.0.1', and 'eml-2.1.0' are individual 'metadataPrefix' parameters, so each requires a separate harvest operation.</p>
<p>Finally, I apologize for not responding to Hannu's query. I will add a copy of this email reply as a comment in Bug <a class="issue tracker-1 status-2 priority-5 priority-highest" title="Bug: design and implement OAI-PMH compliant harvest subsystem (In Progress)" href="https://projects.ecoinformatics.org/ecoinfo/issues/3835">#3835</a> for completeness.</p>
<p>Thanks,<br />Duane</p>
<p>Matt Jones wrote:</p>
<blockquote>
<p>Hi Duane and Ryan,</p>
<p>I've had several requests from different groups who are interested in using the OAI-PMH harvester that you added to Metacat as part of the Dryad project. I was wondering if you could update me on the status of that feature -- what was developed, what is tested, and what has been documented. I noticed that the bug for this enhancement has not been updated, although it does contain a request from Hannu for a status update: <a class="external" href="http://bugzilla.ecoinformatics.org/show_bug.cgi?id=3835">http://bugzilla.ecoinformatics.org/show_bug.cgi?id=3835</a></p>
<p>Also, I was searching for PMH documentation in the Metacat Administrator's Guide, and was unable to find any. Did you document the feature and its configuration elsewhere? If so, it would be great to add it to the admin guide in a new subsection following the current harvester documentation (which I just updated a few days ago to correct some issues).</p>
<p>I would like to release this as a feature in the next Metacat release (1.10) for the DataONE member node implementation. Do you think that is feasible?</p>
<p>Thanks,<br />Matt</p>
</blockquote> Metacat - Bug #3835: design and implement OAI-PMH compliant harvest subsystemhttps://projects.ecoinformatics.org/ecoinfo/issues/3835?journal_id=129812010-05-14T18:40:59ZDuane Costadcosta@lternet.edu
<ul></ul><p>Matt Jones wrote:</p>
<blockquote>
<p>Thanks, Duane. I had missed that document, which seems quite complete after looking it over. Would you be willing to incorporate the technical details of the documentation into the Administrator's Guide? I think most of what you wrote could go in wholesale as it is now.</p>
<p>The one major issue is that we have tried to make configuration pretty easy for people. Maybe we need to add a new screen to the Metacat administration utility that allows people to turn on and off OAI, and possibly set needed properties? For the most part, the default URLs would be fine, and its ok to enable the servlet by default, so this configuration might be a simple checkbox labeled 'Enable OAI-PMH?'. Do you think that would work?</p>
<p>Matt</p>
</blockquote>
<p>Matt,</p>
<p>Yes, with regard to both the Administrator's Guide and the Metacat administration utility, these sound like the best way to go. I'll add this info to the Bugzilla bug (<a class="issue tracker-1 status-2 priority-5 priority-highest" title="Bug: design and implement OAI-PMH compliant harvest subsystem (In Progress)" href="https://projects.ecoinformatics.org/ecoinfo/issues/3835">#3835</a>).</p>
<p>I think this works well in terms of scheduling, too. Between May 1 through August 31, I am working on the second year of the Dryad/LTER project. Ryan or Mark will correct me if I'm wrong, but it seems that adding the finishing touches on the Metacat OAI-PMH work that was started last year would be a worthwhile part of this year's effort (although I should add that the main focus this year is to integrate the LTER controlled vocabulary with other vocabularies using the resources provided by a project called HIVE -- Helping Interdisciplinary Vocabulary Engineering).</p>
<p>Duane</p> Metacat - Bug #3835: design and implement OAI-PMH compliant harvest subsystemhttps://projects.ecoinformatics.org/ecoinfo/issues/3835?journal_id=129822013-03-27T21:24:42ZRedmine Admin
<ul></ul><p>Original Bugzilla ID was 3835</p>