Metacat: Issueshttps://projects.ecoinformatics.org/ecoinfo/https://projects.ecoinformatics.org/ecoinfo/ecoinfo/favicon.ico?14691340362015-01-09T00:20:50ZEcoinformatics Redmine
Redmine Bug #6643 (Closed): EventLog.getD1Report does not correctly map event typeshttps://projects.ecoinformatics.org/ecoinfo/issues/66432015-01-09T00:20:50ZMatt Jonesjones@nceas.ucsb.edu
<p>When reporting log records to DataONE, our EventLog.getD1Report() method does not properly map all of the metacat event types into the DataONE enumeration for event types. By looking at the KNB database, we have 10 distinct event types, which should be mapped onto the dataone event types as follows:</p>
<pre>
METACAT DATAONE
-------- ---------
create => create
delete => delete
insert => create
INSERT => create
read => read
replicate => replicate
synchronization_failed => synchronization_failed
update => update
UPDATE => update
upload => create
</pre>
<p>In particular, metacat only seems to convert 'insert' events to create, whereas it also needs to convert INSERT and upload events, and ensure UPLOAD (all caps) is converted. The SQL snippet uses a case statement, and it is not clear if even 'create' events are making it through unscathed. Based on some counts I have examined comparing metacat tables to DataONE logs, it seems we need more thorough tests that check conversion of all event types.</p>
<pre><code>"case " +<br /> " when event = 'insert' then 'create' " +<br /> " else event " +<br /> "end as event, " +</code></pre> Bug #6141 (Closed): Permission denied to update/publish metadata for group member on Gulfwatchhttps://projects.ecoinformatics.org/ecoinfo/issues/61412013-10-12T00:36:25ZLauren Walkerwalker@nceas.ucsb.edu
<p>Sarah Clark was unable to publish metadata on Gulfwatch even though she is a member of the evos group which has permission type 7 for that guid.</p>
<p>Jing and I could not replicate the problem on dev2 and were able to publish.</p>
<p>Example: <a class="external" href="http://gulfwatch.nceas.ucsb.edu/gulfwatch/d1/mn/v1/meta/df35a.21.31">http://gulfwatch.nceas.ucsb.edu/gulfwatch/d1/mn/v1/meta/df35a.21.31</a></p>
<p>Tomcat log:<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.31"} hits=0 status=0 QTime=1 <br />SEVERE: SystemMetadataEventListener.itemAdded - could not comit the index into the solr server since Solr index doesn't have the information about the id df35a.21.31 which is a component in the resource map resourceMap_df35a.21.31. Metacat-Index can't process the resource map prior to its components.<br />org.apache.solr.client.solrj.SolrServerException: Solr index doesn't have the information about the id df35a.21.31 which is a component in the resource map resourceMapdf35a.21.31. Metacat-Index can't process the resource map prior to its components.<br />INFO: ===================================adding the document df35a.21.31<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.30"} hits=1 status=0 QTime=4 <br />INFO: {delete=[df35a.21.30]} 0 13<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.29"} hits=0 status=0 QTime=7 <br />INFO: {delete=[df35a.21.29]} 0 5<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.28"} hits=0 status=0 QTime=8 <br />INFO: {delete=[df35a.21.28]} 0 5<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.27"} hits=0 status=0 QTime=7 <br />INFO: {delete=[df35a.21.27]} 0 5<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.26"} hits=0 status=0 QTime=2 <br />INFO: {delete=[df35a.21.26]} 0 13<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.25"} hits=0 status=0 QTime=1 <br />INFO: {delete=[df35a.21.25]} 0 13<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.22"} hits=0 status=0 QTime=2 <br />INFO: {delete=[df35a.21.22]} 0 15<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.21"} hits=0 status=0 QTime=7 <br />INFO: {delete=[df35a.21.21]} 0 5<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.20"} hits=0 status=0 QTime=1 <br />INFO: {delete=[df35a.21.20]} 0 13<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.16"} hits=0 status=0 QTime=1 <br />INFO: {delete=[df35a.21.16]} 0 13<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.15"} hits=0 status=0 QTime=7 <br />INFO: {delete=[df35a.21.15]} 0 5<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.14"} hits=0 status=0 QTime=5 <br />INFO: {delete=[df35a.21.14]} 0 5<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.13"} hits=0 status=0 QTime=8 <br />INFO: {delete=[df35a.21.13]} 0 5<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.12"} hits=0 status=0 QTime=7 <br />INFO: {delete=[df35a.21.12]} 0 6<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.11"} hits=0 status=0 QTime=8 <br />INFO: {delete=[df35a.21.11]} 0 5<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.10"} hits=0 status=0 QTime=2 <br />INFO: {delete=[df35a.21.10]} 0 13<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.9"} hits=0 status=0 QTime=1 <br />INFO: {delete=[df35a.21.9]} 0 13<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.8"} hits=0 status=0 QTime=2 <br />INFO: {delete=[df35a.21.8]} 0 12<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.7"} hits=0 status=0 QTime=1 <br />INFO: {delete=[df35a.21.7]} 0 12<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.5"} hits=0 status=0 QTime=1 <br />INFO: {delete=[df35a.21.5]} 0 12<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.4"} hits=0 status=0 QTime=1 <br />INFO: {delete=[df35a.21.4]} 0 12<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.3"} hits=0 status=0 QTime=1 <br />INFO: {delete=[df35a.21.3]} 0 12<br />INFO: [] webapp=null path=/select params={q=id:"df35a.21.31"} hits=0 status=0 QTime=8 <br />INFO: {add=[df35a.21.31]} 0 58<br />INFO: ============================= insert index for the identifier df35a.21.31<br />original pathInfo: /views/metacatui/df35a.21.31<br />original requestURI: /gulfwatch/d1/mn/v1/views/metacatui/df35a.21.31<br />new pathinfo: /views/metacatui/df35a.21.31<br />original pathInfo: /query/solr/fl=id,resourceMap&wt=xml&q=formatType:METADATA+-obsoletedBy:*+resourceMap:*+id:"df35a.21.31" <br />original requestURI: /gulfwatch/d1/mn/v1/query/solr/fl=id,resourceMap&wt=xml&q=formatType:METADATA+-obsoletedBy:*+resourceMap:*+id:%22df35a.21.31%22<br />new pathinfo: /query/solr/fl=id,resourceMap&wt=xml&q=formatType:METADATA -obsoletedBy:* resourceMap:* id:"df35a.21.31" <br />fl=id,resourceMap&wt=xml&q=formatType:METADATA -obsoletedBy:* resourceMap:* id:"df35a.21.31" [edu.ucsb.nceas.metacat.dataone.MNodeService]<br />original pathInfo: /meta/df35a.21.31<br />original requestURI: /gulfwatch/d1/mn/v1/meta/df35a.21.31<br />new pathinfo: /meta/df35a.21.31<br />INFO: [] webapp=null path=/select params={{main(fl=id,resourceMap&q=formatType:METADATA+-obsoletedBy:*+resourceMap:*+id:"df35a.21.31"&wt=xml),extra(fq=(readPermission:"cn%3Devos,o%3DNCEAS,dc%3Decoinformatics,dc%3Dorg")<abbr title="rightsHolder:"cn%3Devos,o%3DNCEAS,dc%3Decoinformatics,dc%3Dorg"">OR</abbr><abbr title="readPermission:"authenticatedUser"">OR</abbr><abbr title="readPermission:"public"">OR</abbr><abbr title="readPermission:"uid%3Dsclark,o%3DNCEAS,dc%3Decoinformatics,dc%3Dorg"">OR</abbr><abbr title="rightsHolder:"uid%3Dsclark,o%3DNCEAS,dc%3Decoinformatics,dc%3Dorg"">OR</abbr>)}} hits=0 status=0 QTime=19 <br />original pathInfo: /meta/df35a.21.31<br />original requestURI: /gulfwatch/d1/mn/v1/meta/df35a.21.31<br />new pathinfo: /meta/df35a.21.31<br />original pathInfo: /isAuthorized/df35a.21.31<br />original requestURI: /gulfwatch/d1/mn/v1/isAuthorized/df35a.21.31<br />new pathinfo: /isAuthorized/df35a.21.31<br />original pathInfo: /publish/df35a.21.31<br />original requestURI: /gulfwatch/d1/mn/v1/publish/df35a.21.31<br />new pathinfo: /publish/df35a.21.31<br />metacat 20131011-13:14:13: [DEBUG]: Updating pid df35a.21.31 [edu.ucsb.nceas.metacat.dataone.D1NodeService]<br />metacat 20131011-13:14:13: [DEBUG]: looking in identifier table for pid df35a.21.31 [edu.ucsb.nceas.metacat.dataone.D1NodeService]<br />metacat 20131011-13:14:13: [DEBUG]: localId: df35a.21.31 [edu.ucsb.nceas.metacat.dataone.D1NodeService]<br />metacat 20131011-13:14:13: [DEBUG]: incremented localId: df35a.21.32 [edu.ucsb.nceas.metacat.dataone.D1NodeService]<br />java.lang.Exception: User uid=sclark,o=NCEAS,dc=ecoinformatics,dc=org does not have permission to update XML Document #df35a.21.31<br />metacat 20131011-13:14:13: [ERROR]: D1ResourceHandler: Serializing exception with code 500: Error inserting or updating document: <?xml version="1.0"?><error>User uid=sclark,o=NCEAS,dc=ecoinformatics,dc=org does not have permission to update XML Document #df35a.21.31</error> [edu.ucsb.nceas.metacat.restservice.D1ResourceHandler]<br />org.dataone.service.exceptions.ServiceFailure: Error inserting or updating document: <?xml version="1.0"?><error>User uid=sclark,o=NCEAS,dc=ecoinformatics,dc=org does not have permission to update XML Document #df35a.21.31</error><br />original pathInfo: /views/metacatui/df35a.21.31<br />original requestURI: /gulfwatch/d1/mn/v1/views/metacatui/df35a.21.31<br />new pathinfo: /views/metacatui/df35a.21.31<br />original pathInfo: /query/solr/fl=id,resourceMap&wt=xml&q=formatType:METADATA+-obsoletedBy:*+resourceMap:*+id:"df35a.21.31" <br />original requestURI: /gulfwatch/d1/mn/v1/query/solr/fl=id,resourceMap&wt=xml&q=formatType:METADATA+-obsoletedBy:*+resourceMap:*+id:%22df35a.21.31%22<br />new pathinfo: /query/solr/fl=id,resourceMap&wt=xml&q=formatType:METADATA -obsoletedBy:* resourceMap:* id:"df35a.21.31" <br />fl=id,resourceMap&wt=xml&q=formatType:METADATA -obsoletedBy:* resourceMap:* id:"df35a.21.31" [edu.ucsb.nceas.metacat.dataone.MNodeService]<br />original pathInfo: /meta/df35a.21.31<br />original requestURI: /gulfwatch/d1/mn/v1/meta/df35a.21.31<br />new pathinfo: /meta/df35a.21.31<br />INFO: [] webapp=null path=/select params={{main(fl=id,resourceMap&q=formatType:METADATA+-obsoletedBy:*+resourceMap:*+id:"df35a.21.31"&wt=xml),extra(fq=(readPermission:"public"))}} hits=0 status=0 QTime=14 <br />original pathInfo: /meta/df35a.21.31<br />original requestURI: /gulfwatch/d1/mn/v1/meta/df35a.21.31<br />new pathinfo: /meta/df35a.21.31<br />original pathInfo: /isAuthorized/df35a.21.31<br />original requestURI: /gulfwatch/d1/mn/v1/isAuthorized/df35a.21.31<br />new pathinfo: /isAuthorized/df35a.21.31<br />metacat 20131011-13:36:02: [ERROR]: D1ResourceHandler: Serializing exception with code 401: CHANGE_PERMISSION not allowed on df35a.21.31 for subject[s]: public; [edu.ucsb.nceas.metacat.restservice.D1ResourceHandler]<br />org.dataone.service.exceptions.NotAuthorized: CHANGE_PERMISSION not allowed on df35a.21.31 for subject[s]: public; <br />original pathInfo: /views/metacatui/df35a.21.31<br />original requestURI: /gulfwatch/d1/mn/v1/views/metacatui/df35a.21.31<br />new pathinfo: /views/metacatui/df35a.21.31<br />original pathInfo: /query/solr/fl=id,resourceMap&wt=xml&q=formatType:METADATA+-obsoletedBy:*+resourceMap:*+id:"df35a.21.31" <br />original requestURI: /gulfwatch/d1/mn/v1/query/solr/fl=id,resourceMap&wt=xml&q=formatType:METADATA+-obsoletedBy:*+resourceMap:*+id:%22df35a.21.31%22<br />new pathinfo: /query/solr/fl=id,resourceMap&wt=xml&q=formatType:METADATA -obsoletedBy:* resourceMap:* id:"df35a.21.31" <br />fl=id,resourceMap&wt=xml&q=formatType:METADATA -obsoletedBy:* resourceMap:* id:"df35a.21.31" [edu.ucsb.nceas.metacat.dataone.MNodeService]<br />original pathInfo: /meta/df35a.21.31<br />original requestURI: /gulfwatch/d1/mn/v1/meta/df35a.21.31<br />new pathinfo: /meta/df35a.21.31<br />INFO: [] webapp=null path=/select params={{main(fl=id,resourceMap&q=formatType:METADATA+-obsoletedBy:*+resourceMap:*+id:"df35a.21.31"&wt=xml),extra(fq=(readPermission:"public"))}} hits=0 status=0 QTime=14 <br />original pathInfo: /meta/df35a.21.31<br />original requestURI: /gulfwatch/d1/mn/v1/meta/df35a.21.31<br />new pathinfo: /meta/df35a.21.31<br />original pathInfo: /isAuthorized/df35a.21.31<br />original requestURI: /gulfwatch/d1/mn/v1/isAuthorized/df35a.21.31<br />new pathinfo: /isAuthorized/df35a.21.31<br />metacat 20131011-13:36:40: [ERROR]: D1ResourceHandler: Serializing exception with code 401: CHANGE_PERMISSION not allowed on df35a.21.31 for subject[s]: public; [edu.ucsb.nceas.metacat.restservice.D1ResourceHandler]</p> Bug #6061 (Closed): Ensure that all packages from metacat API have resource maphttps://projects.ecoinformatics.org/ecoinfo/issues/60612013-09-02T18:03:05ZMatt Jonesjones@nceas.ucsb.edu
<p>Data package downloads no longer have data files in them, even when the data are present in Metacat via an upload from Morpho. This is probably because packages uploaded via the Metacat API (deprecated) do not have an attached resource map file. We need to ensure that these resource map files are systematically included because, without them, the downloads for packages do not include the data. We have gotten several complaints about this since the new MetacatUI interface wen public. TO close this bug:</p>
<p>1) Create a mechanism to create resource maps for all packages as they are uploaded using the Metacat API. This may need to run after all components of the package have been uploaded.<br />2) Ensure that the mechanism creates resource maps for older packages that were uploaded using the Metacat API before the present time</p> Bug #5929 (Closed): replication update action times outhttps://projects.ecoinformatics.org/ecoinfo/issues/59292013-04-30T21:35:33ZMatt Jonesjones@nceas.ucsb.edu
<p>With large database sizes, the replications "update" action times out with normal settings for HTTP timeouts. On DataONE CNs, this action can take more than 4 minutes. I traced this down to the SQL query used to find deleted documents:</p>
<p>select distinct docid from xml_revisions where docid not in (select docid from xml_documents) and server_location = 1;</p>
<p>which takes an excessive amount of time because it materializes a large table. See the DataONE ticket for details (<a class="external" href="https://redmine.dataone.org/issues/3740">https://redmine.dataone.org/issues/3740</a>).</p> Bug #3816 (Closed): Export action doesn't create complete packagehttps://projects.ecoinformatics.org/ecoinfo/issues/38162009-02-10T02:02:05ZShaun Walbridgewalbridge@nceas.ucsb.edu
<p>When using the export action, two bugs are noticeable:</p>
<p>1. Documents with associated data tables have files with the data table names included, but no data contained within:</p>
<p><a class="external" href="http://knb.ecoinformatics.org/knb/metacat?action=export&qformat=nceas&docid=nceas.912.8">http://knb.ecoinformatics.org/knb/metacat?action=export&qformat=nceas&docid=nceas.912.8</a></p>
<p>2. The resulting files are named after the docid and not based on our sensible naming logic (e.g. mbauer.5 instead of nceas.912.8-mbauer.5.2-Benicia_CABE-107-1.txt). Check with Jim / Rick as to their preference. Probably in the context of the package, the metadata docid is superfluous, and the docid-filename form is correct (mbauer.5.2-Benicia_CABE-107.1.txt).</p>