Metacat: Issueshttps://projects.ecoinformatics.org/ecoinfo/https://projects.ecoinformatics.org/ecoinfo/ecoinfo/favicon.ico?14691340362016-11-22T23:10:50ZEcoinformatics Redmine
Redmine Bug #7160 (Closed): replicationPolicy missing numberReplicas and replicationAllowed attributeshttps://projects.ecoinformatics.org/ecoinfo/issues/71602016-11-22T23:10:50ZMatt Jonesjones@nceas.ucsb.edu
<p>While technically optional, the ReplicationPolicy class in SystemMetadata is not useful without the replicationAllowed and numberReplicas attributes. We encountered a bug in the R client where the parser assumed those would be present (<a class="external" href="https://github.com/ropensci/datapack/issues/63">https://github.com/ropensci/datapack/issues/63</a>). We fixed that client side, but it would also be useful to have Metacat generate ReplicationPolicy instances with these attributes set. This is handled in SystemMetadataFactory.getDefaultReplicationPolicy().</p> Support #7076 (Closed): Migrate metacat build to use EML git repohttps://projects.ecoinformatics.org/ecoinfo/issues/70762016-07-28T02:50:38ZMatt Jonesjones@nceas.ucsb.edu
<p>The EML source repository migrated to <a class="external" href="https://github.com/NCEAS/eml">https://github.com/NCEAS/eml</a> . Modify the Metacat build.xml to use git accordingly.</p> Task #6827 (Closed): update SSL configuration directiveshttps://projects.ecoinformatics.org/ecoinfo/issues/68272015-07-29T21:10:42ZMatt Jonesjones@nceas.ucsb.edu
<p>Metacat ships with the metacat-site-ssl file showing how to configure Metacat for SSL support, and references this document in the Admin guide. Certain versions of SSL are now vulnerable, and so we need to ensure that our example configurations and documentation show how to exclude these vulnerabilities. See details below from the security advisory.</p>
<pre>
SSL Version 2 and 3 Protocol Detection (20007)
Synopsis:
The remote service encrypts traffic using a protocol with known weaknesses.
Description:
The remote service accepts connections encrypted using SSL 2.0 and/or SSL 3.0. These versions of SSL reportedly suffer from several cryptographic flaws. An attacker may be able to exploit these flaws to conduct man-in-the-middle attacks or to decrypt communications between the affected service and clients.
NIST has determined that SSL 3.0 is no longer acceptable for secure communications. As of the date of enforcement found in PCI DSS v3.1, any version of SSL will not meet the PCI SSC'S definition of 'strong cryptography'.
See Also:
http://www.schneier.com/paper-ssl.pdf
http://support.microsoft.com/kb/187498
http://www.nessus.org/u?247c4540
https://www.openssl.org/~bodo/ssl-poodle.pdf
http://www.nessus.org/u?5d15ba70
Solution:
Consult the application's documentation to disable SSL 2.0 and 3.0.
Use TLS 1.0 or higher instead.
</pre> Support #6793 (In Progress): Update DOIs from KNB to redirect to view servicehttps://projects.ecoinformatics.org/ecoinfo/issues/67932015-07-02T19:00:00ZMatt Jonesjones@nceas.ucsb.edu
<p>In <a class="issue tracker-2 status-3 priority-2 priority-default closed" title="Feature: Create a new service to update DOI pointers (Resolved)" href="https://projects.ecoinformatics.org/ecoinfo/issues/6530">#6530</a> and <a class="issue tracker-5 status-5 priority-2 priority-default closed" title="Story: Keep DOI registrations current and resolvable (Closed)" href="https://projects.ecoinformatics.org/ecoinfo/issues/6440">#6440</a>, we added features to update DOI registrations, but we still have many originally assigned DOIs that redirect to the raw EML document rather than our landing page for a data set. We need to fix all of the /AA/ DOI registrations in the KNB and ensure they point to the right View service page. For DOIs for metadata, that would be the associated /view url for that DOI. For data files and resource maps, its to the view for the associated metadata. E.g.,</p>
<ul>
<li>Metadata doi:10.5063/AA/nceas.227.15 should redirect to <a class="external" href="https://knb.ecoinformatics.org/#view/doi:10.5063/AA/nceas.227.15">https://knb.ecoinformatics.org/#view/doi:10.5063/AA/nceas.227.15</a></li>
<li>Data doi:10.5063/AA/wtyburczy.30.1 should redirect to the same metadata file <a class="external" href="https://knb.ecoinformatics.org/#view/doi:10.5063/AA/nceas.227.15">https://knb.ecoinformatics.org/#view/doi:10.5063/AA/nceas.227.15</a></li>
</ul>
<p>Also, when a user updates metadata for a package (but doesn't change the data), the DOI redirect for the data will need to be updated to point to the new metadata. Let's verify that this is happening automatically in Metacat.</p> Bug #6677 (Closed): perl registry installation failshttps://projects.ecoinformatics.org/ecoinfo/issues/66772015-02-28T00:52:14ZMatt Jonesjones@nceas.ucsb.edu
<p>On Ubuntu, trying to install the perl registry form the binary metacat release fails following the instructions provided in the Admin guide. This may be only a documentation issue, but in particular, we need to clarify how to install Metacat.pm. The Metacat.pm module is provided in the war file, but it is hidden away in the WEB-INF/lib/Metacat.pm. It is unclear how to install it form the documentation. In section 7.2.1, the documentation says to install it using the standard perl Make utilities, but the source code is not provided in the binary release, so this is not possible.</p> Bug #6662 (Closed): Metacat fails large-file uploadhttps://projects.ecoinformatics.org/ecoinfo/issues/66622015-02-06T06:53:40ZMatt Jonesjones@nceas.ucsb.edu
<p>Metacat seems to have a hard limit set on file upload size, at least for the DataONE MN.create() API. I attempted to call create() on a 4GiB file, which produced the error below in the logs.</p>
<p>Looking into the code, for Metacat 2.4.2, it appears the size limit is hardcoded on line 677 of D1ResourceHandler.java:</p>
<p><code><br />MultipartRequestResolver mrr =<br /> new MultipartRequestResolver(tmpDir.getAbsolutePath(), 1000000000, 0);<br /></code></p>
<p>To fix this, we should set a reasonable size that allows individual files to include typical multi-gigabyte-sized files. At a minimum this should be configurable and not hard-coded.</p>
<p>The produced error was:<br /><code>org.dataone.service.exceptions.ServiceFailure: Could not resolve multipart files: the request was rejected because its size (1000001678) exceeds the configured maximum (1000000000)<br /> at edu.ucsb.nceas.metacat.restservice.D1ResourceHandler.collectMultipartFiles(D1ResourceHandler.java:683)<br /> at edu.ucsb.nceas.metacat.restservice.MNResourceHandler.putObject(MNResourceHandler.java:1381)<br /> at edu.ucsb.nceas.metacat.restservice.MNResourceHandler.handle(MNResourceHandler.java:255)<br /> at edu.ucsb.nceas.metacat.restservice.D1RestServlet.doPost(D1RestServlet.java:84)<br /> at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)<br /> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)<br /> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)<br /> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)<br /> at edu.ucsb.nceas.metacat.restservice.D1URLFilter.doFilter(D1URLFilter.java:48)<br /> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)<br /> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)<br /> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)<br /> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)<br /> at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)<br /> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)<br /> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)<br /> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)<br /> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)<br /> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)<br /> at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193)<br /> at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)<br /> at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)<br /> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<br /> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)<br /> at java.lang.Thread.run(Thread.java:745)<br /></code></p> Bug #6643 (Closed): EventLog.getD1Report does not correctly map event typeshttps://projects.ecoinformatics.org/ecoinfo/issues/66432015-01-09T00:20:50ZMatt Jonesjones@nceas.ucsb.edu
<p>When reporting log records to DataONE, our EventLog.getD1Report() method does not properly map all of the metacat event types into the DataONE enumeration for event types. By looking at the KNB database, we have 10 distinct event types, which should be mapped onto the dataone event types as follows:</p>
<pre>
METACAT DATAONE
-------- ---------
create => create
delete => delete
insert => create
INSERT => create
read => read
replicate => replicate
synchronization_failed => synchronization_failed
update => update
UPDATE => update
upload => create
</pre>
<p>In particular, metacat only seems to convert 'insert' events to create, whereas it also needs to convert INSERT and upload events, and ensure UPLOAD (all caps) is converted. The SQL snippet uses a case statement, and it is not clear if even 'create' events are making it through unscathed. Based on some counts I have examined comparing metacat tables to DataONE logs, it seems we need more thorough tests that check conversion of all event types.</p>
<pre><code>"case " +<br /> " when event = 'insert' then 'create' " +<br /> " else event " +<br /> "end as event, " +</code></pre> Bug #6546 (Resolved): old metacat URIs should redirect to #view servicehttps://projects.ecoinformatics.org/ecoinfo/issues/65462014-05-05T09:25:20ZMatt Jonesjones@nceas.ucsb.edu
<p>Researchers have complained that prior citations on the KNB metacat of the form:</p>
<pre><code><a class="external" href="https://knb.ecoinformatics.org/knb/metacat/nuding.7.6/knb">https://knb.ecoinformatics.org/knb/metacat/nuding.7.6/knb</a></code></pre>
<p>currently return a partially broken response. Instead, we should configure apache redirect all URLs of that for for the KNB to the associated URL for the view service for the data set, in this case:</p>
<pre><code><a class="external" href="https://knb.ecoinformatics.org/#view/doi:10.5063/AA/nuding.7.6">https://knb.ecoinformatics.org/#view/doi:10.5063/AA/nuding.7.6</a></code></pre>
<p>We need to consider how to redirect other skins other than the knb skin that are on that server, such as nrs and obfs.</p> Feature #6196 (Rejected): ORE documents should be generated for new EML records from the Metacat APIhttps://projects.ecoinformatics.org/ecoinfo/issues/61962013-11-01T07:16:13ZMatt Jonesjones@nceas.ucsb.edu
<p>Currently, if a metadata document is uploaded via the Metacat API (action=insert or action=update), it does not contain an ORE document, as that API doesn't support it's upload. Consequently, even if we manually generate ORE maps from the admin control panel, each update of an object fails to create an associated ORE map even if it existed for the prior version. Thus, the data set drops out of any searches including the 'Has data' flag.</p>
<p>To fix this, just as we generate SystemMetadata automatically, we should also automatically generate an ORE document for the new metadata document that links all of the referenced data files into the resource map. This will avoid the need for large batch generation of ORE documents.</p> Bug #6194 (Closed): publish service does not create an ORE documenthttps://projects.ecoinformatics.org/ecoinfo/issues/61942013-11-01T06:59:27ZMatt Jonesjones@nceas.ucsb.edu
<p>When the 'publish' service is called from the GUI in MetacatUI, it should generate an OAI-ORE package so that the linked data and metadata files are all downloadable from the web 'Download' link. This should happen regardless of whether the original document had an ORE or not -- by selecting 'Publish', the user is opting to create a fully fledged data package linking all of the content together.</p> Bug #6193 (Closed): obsoleted versions of objects show up in KNB themehttps://projects.ecoinformatics.org/ecoinfo/issues/61932013-11-01T06:53:47ZMatt Jonesjones@nceas.ucsb.edu
<p>From the MetacatUI page for solson.11.6, I logged in and used the Publish button to generate the DOI doi:10.5063/F1QN64NZ. Now, when searching the KNB skin for the phrase 'olson', both the new document (doi:10.5063/F1QN64NZ) and the now obsoleted version of the document (solson.11.6) show up in search results. Correct behavior would be to only show non-obsoleted versions in search results by default in MetacatUI.</p>
<p>N.B. Of course, the SOLR search itself should be able to search and find obsoleted versions.</p> Bug #6136 (Closed): files left open causes too many file descriptors on OShttps://projects.ecoinformatics.org/ecoinfo/issues/61362013-10-09T20:03:37ZMatt Jonesjones@nceas.ucsb.edu
<p>Metacat writes temp files to disk, and in the process has been failing to close file handles. Over time, especially with operations that touch many files, the number of file handles in use by Metacat increases and eventually exceeds the operating systems hard limit, causing exceptions when Metacat tries to open any additional files. Need to be sure to close all file handles properly after usage.</p> Bug #6086 (Closed): publish service call fails to authenticate properlyhttps://projects.ecoinformatics.org/ecoinfo/issues/60862013-09-06T23:45:08ZMatt Jonesjones@nceas.ucsb.edu
<p>I was trying to issue a DOI for Sarah Olson's data set tonight via curl on the KNB using the KNB node certificate as my identity. Doing this pointed out two issues, the second of which is definitely a bug.</p>
<p>First error<br />---------<br />I ran this:</p>
<pre>
# curl -X PUT -E /var/metacat/certs/urn_node_KNB.pem http://knb.ecoinformatics.org/knb/d1/mn/v1/publish/solson.11.5
<?xml version="1.0" encoding="UTF-8"?>
<error detailCode="1210" errorCode="401" name="InvalidToken">
<description>No session has been provided</description>
</error>
</pre>
<p>That error is traced back to the beginning of MNodeService.update() where sessions are checked. Looking at ezid, I can see that the DOI was reserved successfully, but then the publish fails doing the update() on the object. I had thought that passing in the client cert was sufficient to identify myself and set up a session, but apparently not. Any thoughts on why this didn't work, and what is needed to successfully log in via curl? There may not be a bug here, but rather me using curl incorrectly. Or maybe it should work.</p>
<p>Second error<br />-----------<br />Also, it seems that the EZID mint() call worked even though I wasn't authenticated on metacat, so I tried the same call again without a certificate at all:</p>
<pre>
# curl -X PUT http://knb.ecoinformatics.org/knb/d1/mn/v1/publish/solson.11.5
</pre>
<p>and I got the same error. The new DOI is still reserved on the EZID system despite not being authenticated in Metacat, so there seems to be a bug here in not checking the session before contacting EZID to mint() the identifier.</p> Bug #6061 (Closed): Ensure that all packages from metacat API have resource maphttps://projects.ecoinformatics.org/ecoinfo/issues/60612013-09-02T18:03:05ZMatt Jonesjones@nceas.ucsb.edu
<p>Data package downloads no longer have data files in them, even when the data are present in Metacat via an upload from Morpho. This is probably because packages uploaded via the Metacat API (deprecated) do not have an attached resource map file. We need to ensure that these resource map files are systematically included because, without them, the downloads for packages do not include the data. We have gotten several complaints about this since the new MetacatUI interface wen public. TO close this bug:</p>
<p>1) Create a mechanism to create resource maps for all packages as they are uploaded using the Metacat API. This may need to run after all components of the package have been uploaded.<br />2) Ensure that the mechanism creates resource maps for older packages that were uploaded using the Metacat API before the present time</p> Task #5994 (New): Create REST API for accessing statisticshttps://projects.ecoinformatics.org/ecoinfo/issues/59942013-05-25T01:16:03ZMatt Jonesjones@nceas.ucsb.edu
<p>For objects, users, packages, nodes, etc.</p>