Metacat: Issueshttps://projects.ecoinformatics.org/ecoinfo/https://projects.ecoinformatics.org/ecoinfo/ecoinfo/favicon.ico?14691340362017-11-14T00:24:20ZEcoinformatics Redmine
Redmine Task #7227 (Resolved): metacat.EventLog.getReport should downgrade a log levelhttps://projects.ecoinformatics.org/ecoinfo/issues/72272017-11-14T00:24:20ZJing Taotao@nceas.ucsb.edu
<p>Details should be found at:<br /><a class="external" href="https://redmine.dataone.org/issues/8110">https://redmine.dataone.org/issues/8110</a></p> Bug #7221 (Resolved): Remove the duplicated the log4j file in Metacathttps://projects.ecoinformatics.org/ecoinfo/issues/72212017-10-20T22:36:49ZJing Taotao@nceas.ucsb.edu
<p>Currently we have to copies of log4j files in Metacat. It causes the problem in tomcat 8 in linux. We need to remove the old version one.</p> Bug #7220 (Resolved): ORCID links in View Service output don't open in a new tabhttps://projects.ecoinformatics.org/ecoinfo/issues/72202017-10-20T22:22:07ZBryce Mecummecum@nceas.ucsb.edu
<p>Generally, external links on the View Service page open in a new tab, utilizing the HTML5 target attribute. I just noticed that the links to ORCID profiles do not. This was my mistake back when I added the feature. We should fix this!</p>
<p>Note: This applies only to EML documents at the moment.</p> Bug #7214 (Closed): Metacat overwrites the deployDir propertyhttps://projects.ecoinformatics.org/ecoinfo/issues/72142017-10-05T00:51:02ZChris Jonescjones@nceas.ucsb.edu
<p>On a fresh installation of Metacat, the <code>application.deployDir</code> property value gets overwritten. This is undesirable for Docker deployments where the Metacat Admin UI is not used for configuration, but rather the <code>metacat.properties</code> file is pre-configured for deployment. Fix this. It looks to be happening in the <code>PropertiesAdmin.configureProperties()</code> method:<br /><pre>
PropertyService.setPropertyNoPersist("application.deployDir",
SystemUtil.discoverDeployDir(request));
</pre></p>
<p>Look at the consequences of not calling <code>discoverDeployDir()</code> and perhaps remove this. Needs review.</p> Bug #7211 (Closed): Metacat sets a wrong default value for the deploy path during the totally fre...https://projects.ecoinformatics.org/ecoinfo/issues/72112017-10-03T16:01:43ZJing Taotao@nceas.ucsb.edu
<p>Matt reported: I’ve noticed that when run from scratch, the metacat /admin utility seems to tack the /webapps/metacat onto the deployDir property, and I always have to change it. SO I think that is an admin utility bug on a totally clean deployment (i.e., with no backup directory in place).<br />so, this arises with clean deploys from docker and the like.</p> Bug #7207 (Rejected): Requests to view service are not properly encoding identifiershttps://projects.ecoinformatics.org/ecoinfo/issues/72072017-08-22T16:18:35ZLauren Walkerwalker@nceas.ucsb.edu
<p>From Dave Vieglais:</p>
<p>All requests to DataONE services must properly URL path encode identifiers that are included with GET requests.</p>
<p>For example, when viewing a search result for the identifier:</p>
<p><a class="external" href="http://dx.doi.org/10.5061/dryad.j1828/2?ver=2017-08-19T07:36:06.033-04:00">http://dx.doi.org/10.5061/dryad.j1828/2?ver=2017-08-19T07:36:06.033-04:00</a></p>
<p>the view request fails with a 404 error since the identifier is not being encoded:</p>
<p><a class="external" href="https://search.dataone.org/cn/v2/views/metacatui/http://dx.doi.org/10.5061/dryad.j1828?ver=2017-08-19T07:35:55.841-04:00">https://search.dataone.org/cn/v2/views/metacatui/http://dx.doi.org/10.5061/dryad.j1828?ver=2017-08-19T07:35:55.841-04:00</a></p>
<p>Properly encoding the request:</p>
<p><a class="external" href="https://search.dataone.org/cn/v2/views/metacatui/http%3A%2F%2Fdx.doi.org%2F10.5061%2Fdryad.j1828%3Fver%3D2017-08-19T07%3A35%3A55.841-04%3A00">https://search.dataone.org/cn/v2/views/metacatui/http%3A%2F%2Fdx.doi.org%2F10.5061%2Fdryad.j1828%3Fver%3D2017-08-19T07%3A35%3A55.841-04%3A00</a></p>
<p>yields the expected HTTP 200 response.</p> Task #7206 (Resolved): Change the behavior to archive the older version of the resource map objec...https://projects.ecoinformatics.org/ecoinfo/issues/72062017-08-17T05:15:31ZJing Taotao@nceas.ucsb.edu
<p>Do you know if Morpho is by default archiving the resource map when a new version of a data package is pushed to KNB? <br />I am asking because I realized that when I try to go back to previous versions of our SNAPP packages, it seems that all the previous resourceMaps were archived (thanks for @peter for helping me to understand that) restricting the access to previous versions to the EML only. Here as example: <a class="external" href="https://knb.ecoinformatics.org/#view/karakoenig.46.14">https://knb.ecoinformatics.org/#view/karakoenig.46.14</a> as the lateset and <a class="external" href="https://knb.ecoinformatics.org/#view/karakoenig.46.12">https://knb.ecoinformatics.org/#view/karakoenig.46.12</a> as a previously shared version with the PIs.</p> Bug #7201 (Closed): Some DataONE service packages not being reportedhttps://projects.ecoinformatics.org/ecoinfo/issues/72012017-06-27T21:11:47ZPeter Slaughterslaughter@nceas.ucsb.edu
<p>The DataONE 'node' service requests a member node to provide a list of service packages that it supports, for example <a class="external" href="https://knb.ecoinformatics.org/knb/d1/mn/v2/node">https://knb.ecoinformatics.org/knb/d1/mn/v2/node</a><br />will list the implemented packages:<br />...<br /><services><br /><service name="MNCore" version="v1" available="true"/><br /><service name="MNCore" version="v2" available="true"/><br /><service name="MNRead" version="v1" available="true"/><br /><service name="MNRead" version="v2" available="true"/><br /><service name="MNAuthorization" version="v1" available="true"/><br /><service name="MNAuthorization" version="v2" available="true"/><br /><service name="MNStorage" version="v1" available="true"/><br /><service name="MNStorage" version="v2" available="true"/><br /><service name="MNReplication" version="v1" available="true"/><br /><service name="MNReplication" version="v2" available="true"/><br /></services><br />...</p>
<p>However, it appears that not all service packages are being reported, as KNB supports the 'getPackage' service that is part of "MNPackage", which does not appear.<br />Also, "MNView" and "MNQuery" do not appear.</p>
<p>The full list of course, is here: <a class="external" href="https://purl.dataone.org/architecture-dev/apis/MN_APIs.html">https://purl.dataone.org/architecture-dev/apis/MN_APIs.html</a></p> Bug #7195 (Resolved): LDAP-based group authorization is failinghttps://projects.ecoinformatics.org/ecoinfo/issues/71952017-05-26T17:44:46ZChris Jonescjones@nceas.ucsb.edu
<p>In calls to <code>MNRead.getSystemMetadata()</code> where the user logged in is a member of a group, and the group is listed in the <code>SystemMetadata.AccessPolicy</code> with read, write, and changePermission permissions, authorization to read the private object is not allowed. In <code>D1NodeService.getSystemMetadata()</code>, we call <code>isAuthorized()</code>, which in turn calls <code>userHasPermission()</code>. Recent changes to this code allows for DataONE group authorization to work. However, group DNs defined in the <code>dc=ecoinformatics,dc=org</code> LDAP tree that are listed in the <code>AccessPolicy</code> and stored Metacat look to not be expanded to their individual group members when comparing DNs. On lines 1229 to 1255 of <code>D1NodeService</code>, we call:<br /><pre>
if (accessRule.getSubjectList().contains(s)) {
logMetacat.debug("Access rule contains subject: " + s.getValue());
for (Permission p: accessRule.getPermissionList()) {
logMetacat.debug("Checking permission: " + p.xmlValue());
expandedPermissions = expandPermissions(p);
allowed = expandedPermissions.contains(permission);
if (allowed) {
logMetacat.info("Permission granted: " + p.xmlValue() + " to " + s.getValue());
break search; //label break
}
}
}
</pre><br />It looks like the call to <code>accessRule.getSubjectList()</code> is not expanding the the list of subjects if the subject DN is a group from the locally configured Metacat auth store (LDAP or file store).</p>
<p>An example is in the KNB: <code>kengmiller.14.15</code>. This object has an access rule allowing <code>cn=snapp,o=NCEAS,dc=ecoinformatics,dc=org</code> to r/w/chP. Members of the group can not read the object, much less modify it. Jing, I've added you to the SNAPP group for troubleshooting this.</p>
<p>Change <code>userHasPermission()</code> to correctly expand all groups, and write a unit test to exercise group-based authorization from auth file, LDAP, or DataONE group definitions.</p> Bug #7192 (Resolved): MN.update method doesn't check the checksumhttps://projects.ecoinformatics.org/ecoinfo/issues/71922017-05-22T15:49:43ZJing Taotao@nceas.ucsb.edu
<p>The MN.update doesn't check the checksum - it doesn't compute the checksum of the coming object and compare the computed value with the one in the system metadata. We need to check as the create method does.</p> Bug #7188 (Resolved): MNodeService.replicate() is failinghttps://projects.ecoinformatics.org/ecoinfo/issues/71882017-05-11T14:19:25ZChris Jonescjones@nceas.ucsb.edu
<p>Laura Moyers reported that she is seeing many failed replication attempts in the Coordinating Node index. In particular, KNB, GOA, UIC, ARCTIC, mnUCSB1, and mnORC1 are all affected, and are all running Metacat.</p>
<p>After looking at catalina.out on the MNs, we're seeing errors in <code>MNodeService.replicate()</code>:<br /><pre>
20170508-06:59:14: [ERROR]: Error computing checksum on replica: mark/reset not supported [edu.ucsb.nceas.metacat.dataone.MNodeService]
</pre></p>
<p>Here's the number of requests and failures<br /><pre>
host requests failures failures_since
-----------------------------------------------------------
mn-orc-1 145 25 20170511-01:23:48
mn-ucsb-1 105 56 20170508-06:59:14
mn-unm-1 0 0 -
knb 71 28 20170509-16:57:53
uic no log access
</pre></p>
<p>I'm pretty sure the failures represent 100% of the requests since the failures began, but we'd need to confirm this. Basically, MN replication looks to be entirely broken in Metacat.</p>
<p>The error reported above comes from line 866 of <code>MNodeService.java</code>, where the checksum of the bytes of the object from the source MN (to be replicated) is calculated. Once the checksum is calculated, we call <code>object.reset()</code> on the input stream so it can be read again when writing to disk. This is throwing the exception above.</p>
<p>So what's changed? The last changes regarding the <code>InputStream</code> was that Jing wrapped the calls in a <code>try{ } finally { }</code> block in order to ensure the input stream gets closed after use to prevent memory leaks. This doesn't seem like an issue at all, although the <code>finally{ }</code> block could have been used in the existing <code>try { }</code> block instead of having three levels of <code>try</code> nesting. This seems inconsequential though.</p>
<p>The other change is that <code>d1_libclient_java</code> is now using the Apache Commons IO <code>AutoCloseInputStream</code>. Looking at the documentation there, it seems to delegate to the underlying input stream implementation. We know that not all input streams support the <code>mark()</code> method and therefore can't be <code>reset()</code>, which is why we call <code>markSupported()</code> before attempting to calculate the checksum. So why is <code>markSupported()</code> succeeding, but then <code>reset()</code> is failing after reading the input stream? It seems like we need to track this down between the interaction of <code>MNodeService</code> and <code>MultipartMNode.getReplica()</code>.</p> Bug #7187 (Resolved): Set file names correctly when reading objects from Metacathttps://projects.ecoinformatics.org/ecoinfo/issues/71872017-05-10T15:05:57ZChris Jonescjones@nceas.ucsb.edu
<p>We now store file names in <code>SystemMetadata.fileName</code>. In <code>MetacatHandler.readFromMetacat()</code>, we are currently generating a file name based on the <code>docid</code>, and setting it into the <code>Content-Disposition</code> header. Change <code>generateOutputName()</code> to use <code>SystemMetadata.fileName</code> when it's available, and fall back to generating a file name.</p> Bug #7186 (Resolved): listObjects fails if there is pid with a white space in the listhttps://projects.ecoinformatics.org/ecoinfo/issues/71862017-05-02T00:47:57ZJing Taotao@nceas.ucsb.edu
<p><a class="external" href="https://redmine.dataone.org/issues/8085#change-28820">https://redmine.dataone.org/issues/8085#change-28820</a></p> Bug #7185 (Resolved): EML SAX parser will not check if the user has the all permission on data ob...https://projects.ecoinformatics.org/ecoinfo/issues/71852017-05-01T19:11:47ZJing Taotao@nceas.ucsb.edu
<p>In the old Metacat API, the data objects' access rules are controlled by the EML object which describes it. So when the eml parser detects the newer version eml is trying to change the access rules of data objects, it need to verify the user has the ALL permission on the data objects. Otherwise, any users can change the data objects' access rule.<br />However, on DataONE API, data objects' access rules are specified in the system metadata associated with data objects. The access rules on the eml object doesn't control them any more. So we don't need to check it.</p> Bug #7184 (Resolved): The count number is -1 when the expandRightsHolder method lists the subjectshttps://projects.ecoinformatics.org/ecoinfo/issues/71842017-04-26T22:41:36ZJing Taotao@nceas.ucsb.edu
<p>Please see details on:<br /><a class="external" href="https://redmine.dataone.org/issues/8059">https://redmine.dataone.org/issues/8059</a></p>