Metacat

I almost forgot this bug! Thanks for input it. Can you join to irc on tomorrow?

it seems we need to correct local documents in lter. it has no relation to the release. So i postpone it to 1.9

Jing and I investigated further. Here's what we've learned so far:

(1) Many of the EML documents that fail to be replicated are old revisions. In many cases the newer revisions have successfully been replicated from LTER to KNB. This could indicate that something that was broken in Metacat when the older EML revision was inserted into LTER has since been fixed in a newer version of Metacat when the newer revision of the EML document was inserted. Or it could mean that something in the newer revision of the EML document itself changed to overcome the problem (less likely, we think).

I don't have an exact percentage of which EML documents that fail to replicate are old revisions versus current revisions. It would take a lot of time to do this analysis. A sampling of EML documents that cannot be replicated seems to indicate that many are old revisions. This is good, since it's less important for old revisions to be replicated (ideally, of course, we would like everything to be replicated for completeness). On the other hand, it's bad that these replication failures continue to be repeated with every timed replication, using up resources on both the LTER and KNB metacat servers.

(2) We compared an EML document in LTER that fails to be replicated to its original source document (in this case at the CDR LTER site). We found differences such as:

|Exp|Plot|Subplot|taxon|Species|Seedlings <= Labels

|Exp|Plot|Subplot|taxon|Species|Seedlings <= Labels

which involve XML character entities. In this example, the original document contained the character entity '<', while the Metacat document contained the literal '<' character, causing it to fail XML validation, which explains why it cannot be replicated from LTER to KNB.

Jing thinks that the differences were introduced by a known bug which has since been resolved in Metacat 1.8.1:

http://bugzilla.ecoinformatics.org/show_bug.cgi?id=2517

(3) Possible resolutions. Jing suggests manual changes to the LTER Metacat database could fix these problems. For example, find all literal instances of '<' in the nodedata field of xml_nodes and xml_nodes_revisions and change them to the character entity '<'. (Likewise for '>' and perhaps other characters, but need to be careful about '&'.) We both agree that any changes to the content of the nodedata field would have to be done with extreme caution because:
(a) changing nodedata is a big deal, it means you're changing the actual content of the document;
(b) you could easily end up making things worse instead of better!

Duane
(

Some additional analysis of the replication errors:

A total of 1016 EML documents fail to replicate from LNO to KNB.

668 of the documents that fail to replicate are old revisions (i.e. in xml_revisions table).

348 of the documents that fail to replicate are current revisions (i.e. in xml_documents table).

Of the 1016 EML documents that fail to replicate from LNO to KNB, we identified 611 that are directly related to literal characters in inline data files that resulted in invalid XML. There were two cases:

1. literal '&' character in inline data, e.g. :

'Mosses & Lichens'

2. literal '<' character in inline data, e.g. :

'|Exp|Plot|Subplot|taxon|Species|Seedlings <= Labels

Of these 611 instances, 254 were in current EML revisions and 357 were in old EML revisions. All 611 documents originated from the Cedar Creek (CDR) LTER and all document identifiers began with the string pattern 'knb-lter-cdr'. It appears that the original CDR documents contained the appropriate XML character entities but these were converted to literal '&' and '<' characters during insertion into Metacat (possibly related to bugs #2517 and #2797).

We have repaired all 611 of these documents using the following procedure:

1. For safety, we created a backup of the 'data/knb/inlinedata' directory.

2. In the original 'data/knb/inlinedata' directory, we ran the following Perl commands:

(a) perl -pi -e 's/ \& / \& /g' *

(b) perl -pi -e 's/</&lt;/g' *

Note that the first Perl command included a leading and trailing space character in the string pattern. This was a precautionary measure to avoid replacing the leading ampersand character in existing character entities (which could result in something incorrect like '&lt;'); we were able to use this pattern because all instances of literal ampersands that needed replacement were of the form ' & '.

This repair procedure has two benefits:

1. There were 254 CDR documents that appeared in Metacat search results in the LNO Metacat, but they could not be read because they generated invalid XML. All 254 of these documents should now be able to be read.

2. All 611 CDR documents (254 current, 357 old) that previously could not be replicated to knb.ecoinformatics.org should now replicate successfully. The next timed replication from LNO to KNB is expected to run this evening (5/21/2008) so we should be able to verify this tomorrow morning.

We have reduced the number of EML documents that fail to replicate from the original 1016 down to 94. This was accomplished by:

1. Repair of inline data for 611 CDR documents as described in the previous comment. Note, however, that although these documents were successfully replicated to KNB, they also needed to be repaired at KNB because the same Metacat bug that corrupted them when they were inserted at LTER has also corrupted them at KNB.

2. Purge of 311 corrupted documents from the LTER metacat. All 311 of these documents were old revisions, not current revisions. (LTER Information Managers at the affected sites were given prior notice of the plan to purge corrupted old revisions and there were no objections.)

Some ongoing concerns include:

1. There are still 94 documents that fail to replicate. These are all current revisions. In cases where we know how a document can be repaired, we will work with the LTER Information Manager to correct the problem, as time allows.

2. There are 188 replication errors because each of the 94 documents fails to replicate twice. This replication bug (#3304) has already been fixed by Jing, but the fix has not yet been deployed to the KNB metacat.

3. Timed replication requires almost eight hours to complete. (Of course, this should be cut in half after the bug fix for #3304 is deployed.) We'd like to understand better why it takes so long to replicate 188 documents.

4. The 611 CDR documents replicated to KNB need to be repaired by fixing their inline data.

Follow-up to the previous comment of 2008-06-05:

1. There are still 94 documents that fail to replicate. These are all current
revisions. In cases where we know how a document can be repaired, we will work
with the LTER Information Manager to correct the problem, as time allows."

-- This is an ongoing LTER issue so it doesn't need to be tracked in this bug.

2. There are 188 replication errors because each of the 94 documents fails to
replicate twice. This replication bug (#3304) has already been fixed by Jing,
but the fix has not yet been deployed to the KNB metacat.

-- Replication log output indicates that replication is now running once instead of twice, verifying that Jing's bug fix is now deployed.

3. Timed replication requires almost eight hours to complete. (Of course, this
should be cut in half after the bug fix for #3304 is deployed.) We'd like to
understand better why it takes so long to replicate 188 documents.

-- Timed replication now completes in less than ninety minutes.

4. The 611 CDR documents replicated to KNB need to be repaired by fixing their
inline data.

-- Jing will open a separate bug to track this issue.

This bug can now be closed out.

Original Bugzilla ID was 3296