/docs/user/harvester.html - Annotate - Metacat - Ecoinformatics Redmine

metacat/docs/user/harvester.html @ 2294

-            costa
+<!--
   * harvester.html
+  *
   *      Authors: Duane Costa
   *    Copyright: 2004 Regents of the University of California and the
   *               National Center for Ecological Analysis and Synthesis,
   *               and the University of New Mexico.
   *  For Details: http://www.nceas.ucsb.edu/
   *      Created: 2004 April 9
   *      Version:
   *    File Info: '$ '
+  *
+  *
 -->
 <HTML>
 <HEAD>
 <TITLE>Metacat Harvester</TITLE>
 <link rel="stylesheet" type="text/css" href="@docrooturl@default.css">
 </HEAD>
 <BODY>
   <table width="100%">
     <tr>
       <td class="tablehead" colspan="2">
         <p class="label">Metacat Harvester</p>
       </td>
       <td class="tablehead" colspan="2" align="right">
         <a href="./properties.html">Back</a> |
         <a href="./metacattour.html">Home</a> |
         <a href="./unimplem.html">Next</a>
       </td>
     </tr>
   </table>
   <h4>Introduction</h4>
 The Metacat Harvester (henceforth referred to as "Harvester") is a
 program that automates the retrieval of EML documents from one or more sites
 and their subsequent upload (insert or update) to Metacat. Harvester uses pull
 technology to retrieve and upload documents to Metacat on a regularly
 scheduled basis.
 <P>
 Although Harvester is included with a Metacat installation (beginning with
 Metacat version 1.4.0), it is an extention to Metacat's functionality
 that may be used optionally.
 </P>
   <h4>Definitions</h4>
 The following table defines a number of terms that are useful in discussing
 Harvester and its features.
   <br><br>
   <table border="1">
     <tr>
       <td><b>Term</b></td>
       <td><b>Definition</b></td>
     </tr>
     <tr>
       <td>Harvester</td>
       <td>The Harvester program, a Java application that is bundled with the
           Metacat distribution. When a user installs Metacat on a system,
           the Harvester program is automatically included in the
           installation.
       </td>
     </tr>
     <tr>
       <td>Harvester Administrator</td>
       <td>The individual who installs and manages Harvester. Typically, this
           would be the same individual who installs and manages Metacat at a
           given installation.
       </td>
     </tr>
     <tr>
       <td>Harvest Site</td>
       <td>A location from which Harvester can retrieve EML documents. A given
           Harvester can retrieve documents from any number of different
           Harvest Sites.
       </td>
     </tr>
     <tr>
       <td>Harvest</td>
       <td>The act (by Harvester) of visiting a Harvest Site, retrieving a
           number of EML documents, and inserting or updating the documents to
           Metacat.
       </td>
     </tr>
     <tr>
-            costa
+      <td><a name="HarvestList" >Harvest List</a></td>
-            costa
+      <td>An XML document that lists a set of EML documents to be harvested. The
           Harvest List must conform to an XML Schema,
           <a href="../../lib/harvester/harvestList.xsd">harvestList.xsd</a>.
       </td>
     </tr>
     <tr>
       <td>Site Contact</td>
       <td>The individual at a particular Harvest Site who registers with
           Harvester, composes a Harvest List, and periodically prepares
           the site's EML documents for retrieval and upload to Metacat.
       </td>
     </tr>
     <tr>
       <td>Harvest List URL</td>
       <td>A URL to the Harvest List, as specified by the Site Contact.
           Each Harvest Site corresponds to a Harvest List URL. Harvester
           uses the URL to locate and read a site's Harvest List.
       </td>
     </tr>
     <tr>
       <td>Document URL</td>
       <td>A URL to an EML document, as specified in the Harvest List.
           The Harvest List may contain any number of Document URLs. Each
           Document URL provides a locator to a document to be harvested.
       </td>
     </tr>
     <tr>
       <td>Harvester Registration Page</td>
       <td>A web page that provides a means for a Site Contact
           to register with Harvester to schedule regular harvests from the
           site. Registration involves logging in and then specifying various
           settings for the Harvest Site, such as the Harvest List URL, the
           harvest frequency, and the email address of the Site Contact.
       </td>
     </tr>
   </table>
   <h4>Managing Harvester</h4>
   Harvester is managed by the Harvester Administrator. Typically, the same
   individual who manages a Metacat server would also act as the Harvester
   Administrator. The responsibilities of the Harvester Administrator include:
     <ul>
       <li><a href="#Configuring Harvester">Configuring Harvester</a></li>
       <li><a href="#Running Harvester">Running Harvester</a></li>
       <li><a href="#Reviewing Harvester">Reviewing Harvester reports to
       the Harvester Administrator</a></li>
     </ul>
   <h5><a name="Configuring Harvester">Configuring Harvester</a></h5>
   <p>Harvester must be configured to interact with a working Metacat
      installation. Thus, a Metacat installation that has been properly
      configured and installed is a pre-requisite to running Harvester.
      Additionally, Harvester has a number of settable properties that
      control its behavior. All Harvester configuration information is managed
      in a single file,
-            costa
+     <a href=../../lib/metacat.properties>metacat.properties</a>,
-            costa
+     located at:
-            costa
+  <pre>      METACAT_HOME/lib/metacat.properties</pre>
-            costa
+     where METACAT_HOME is the top-level directory that Metacat is
-            costa
+     installed in.
-            costa
+  </p>
-            costa
+  <p>Harvester properties are grouped together in
      <a href=../../lib/metacat.properties>metacat.properties</a>, beginning
      after the comment line:
   <pre><code>      # Harvester properties</code></pre>
   </p>
-            costa
+  <p>The Harvester Administrator should edit
-            costa
+     <a href=../../lib/metacat.properties>metacat.properties</a>,
      setting appropriate values for the <code><b>harvesterAdministrator</b></code>
      property, the <code><b>smtpServer</b></code> property, and possibly other
      properties. The following table is a summary of each property and its function.
-            costa
+  </p>
   <table border="1">
     <tr>
       <td><b>Property</b></td>
       <td><b>Description</b></td>
       <td><b>Possible or default value</b></td>
     </tr>
     <tr>
       <td>connectToMetacat</td>
       <td>This property determines whether Harvester should connect to
           Metacat to upload documents. It should be set to <code>true</code>
           under most circumstances. Setting this property to <code>false</code>
           can be useful for testing whether Harvester is able to retrieve
           documents from a site without actually connecting to Metacat to
           upload the documents.</td>
       <td><code>true</code> | <code>false</code><br>
           Default: <code>true</code>
     </tr>
     <tr>
       <td>delay</td>
       <td>The number of hours that Harvester will wait before beginning its
           first harvest. For example, if Harvester is run at  1:00 p.m., and
           the delay is set to 12, Harvester will begin its first harvest at
 :00 a.m.</td>
       <td>Default: 0</td>
     </tr>
     <tr>
       <td>harvesterAdministrator</td>
       <td>The email address of the Harvester Administrator. Harvester will
           send email reports to this address after every harvest.
       </td>
       <td>An email address</td>
     </tr>
     <tr>
       <td>logPeriod</td>
       <td>The number of days that Harvester should retain log entries of harvest
           operations in the database. Harvester log entries record information
           such as which documents were harvested, from which sites, and
           whether any errors were encountered during the harvest. Log entries
           older than <code>logPeriod</code> number of days are purged from the
           database at the end of each harvest.</td>
       <td>Default: 90</td>
     </tr>
     <tr>
       <td>maxHarvests</td>
       <td>The maximum number of harvests that Harvester should execute before
           shutting down. When the Harvester program is executed, it will
           continue running until it has executed <code>maxHarvests</code>
           number of harvests and then the program will terminate.</td>
       <td>Default: 30</td>
     </tr>
     <tr>
       <td>period</td>
       <td>The number of hours between harvests. Harvester will run a new
           harvest every <code>period</code> number of hours, until the
           <code>maxHarvests</code> number of harvests have been run.</td>
       <td>Default: 24</td>
     </tr>
     <tr>
       <td>smtpServer</td>
       <td>The SMTP server that Harvester uses for sending email messages
           to the Harvester Administrator and to Site Contacts.</td>
       <td>A host name, for example: <code>somehost.institution.edu</code>
           <br><br>
           Default: <code>localhost</code>
           <br><br>
           Note that the default value will only work if the Harvester
           host machine has been configured as a SMTP server.
       </td>
     </tr>
     <tr>
       <td>Harvester Operation Properties (GetDocError, GetDocSuccess, etc.)</td>
       <td>This group of properties is used by Harvester to report information
           about the operations it performs for inclusion in log
           entries and email messages. Under most circumstances the values
           of these properties should not be modified.</td>
       <td>&nbsp;</td>
     </tr>
   </table>
   <br>
   <h5><a name="Running Harvester">Running Harvester</a></h5>
   After Harvester has been appropriately
   <a href="#Configuring Harvester">configured</a>,
   it can be run as follows:
   <ol>
   <li>Open a system command window or terminal window.</li>
   <li>Set the METACAT_HOME environment variable to the value of the Metacat
       installation directory. Some examples follow:
       <ul>
         <li>On Windows:
         <pre>set METACAT_HOME=C:\somePath\metacat</pre></li>
         <li>On Linux/Unix (bash shell):
         <pre>export METACAT_HOME=/home/somePath/metacat</pre></li>
       </ul>
   <li>cd to the following directory:
       <ul>
         <li>On Windows:
         <pre>cd %METACAT_HOME%\lib\harvester</pre></li>
         <li>On Linux/Unix:
         <pre>cd $METACAT_HOME/lib/harvester</pre></li>
       </ul>
   <li>Run the appropriate Harvester shell script, as determined by the
       operating system:
       <ul>
         <li>On Windows:
         <pre>runHarvester.bat</pre></li>
         <li>On Linux/Unix:
         <pre>sh runHarvester.sh</pre></li>
       </ul>
   </li>
   </ol>
   <p>The Harvester application will start executing. It will begin its first
   harvest after <code><b>delay</b></code> number of hours (as specified in the
-            costa
+  <a href=../../lib/metacat.properties>metacat.properties</a>
-            costa
+  file). The application will continue running a new harvest every
   <code><b>period</b></code> number of hours until a <code><b>maxHarvests</b></code>
   number of harvests have been completed.
   </p>
   <h5><a name="Reviewing Harvester">
   Reviewing Harvester Reports to the Harvester Administrator</a></h5>
   <P>
   After every harvest, Harvester will send an email report to the Harvester
   Administrator detailing the operations that were performed during the
   harvest. The report will contain information about each of the Harvest Sites
   that were harvested from, such as which EML documents were
   harvested and whether any errors were encountered.
   </P>
   <p>
   The harvest report will contain a list of log entries, where each log entry
   describes an operation that was performed by Harvester. Log entries that
   show a status value of 1 indicate that an error occurred during the
   operation, while those that show a status value of 0 indicate that the
   operation was completed successfully.
   </p>
   <P>The Harvester Administrator should review the report, paying particularly
   close attention to any errors that are reported and to the accompanying error
   messages that are displayed. When errors are reported at
   a particular site, the Harvester Administrator should contact the Site
   Contact to determine the source of the error and its resolution. See
   <a href=#Reviewing>Reviewing Harvester Reports to the Site Contact</a> for a
   description of common sources of errors at a Harvest Site.
   </P>
   <p>Errors that are independent of a particular site may indicate a problem
   with Harvester itself, Metacat, or the database connection. Refer to the
   error message to determine the source of the error and its resolution.
   </p>
   <h4>Managing a Harvest Site</h4>
   A Harvest Site is managed by a Site Contact.
   The responsibilities of a Site Contact fall into the following categories:
     <ul>
       <li><a href=#Registering>Registering with Harvester</a></li>
       <li><a href=#Composing>Composing a Harvest List</a></li>
       <li><a href=#Preparing>Preparing EML Documents for harvest</a></li>
       <li><a href=#Reviewing>Reviewing Harvester reports to the Site Contact</a></li>
     </ul>
     <h5><a name="Registering">Registering with Harvester</a></h5>
   <p>
   A Site Contact registers a site with Harvester by logging in to the
   Harvester Registration page and entering several items of information
   that Harvester needs to know about the site.
   </p>
   <ol>
     <li>Logging in to the Harvester Registration Page
   <p>
   The Harvester Registration page is accessed from Metacat. For example, if
   the Metacat server that you wish to register with resides at the following
   URL:
   <pre>  http://somehost.somelocation.edu:8080/knb/index.jsp</pre>
   then the Harvester Registration page would be accessed at:
-            costa
+  <pre>  http://somehost.somelocation.edu:8080/knb/style/skins/knb/harvesterRegistrationLogin.html</pre>
-            costa
+  </p>
   <p>
   After bringing up this page in your browser, login to your Metacat account
-            costa
+  by entering your username, organization, and password. For example:
       <table bgcolor="#ffffff" border="0" cellpadding="2" width='100%' >
         <tr >
           <td colspan=3 align=center >&nbsp;</td>
         </tr>
         <tr >
           <td colspan=3 align=center >
             <font face=verdana size=1%>
               <b>Please  Enter Username, Organization, and Password </b>
             </font>
           </td>
         </tr>
         <tr>
           <td width='10%'> &nbsp;</td>
           <td width="25%" bgcolor="#4682b4">
             <p align="center">
             <font color="white" face=verdana size=2%>
             <b>Username</b>
             </font>
           </td>
           <td><p><input type="text" name="uid" value="jdoe" maxlength="100" size="28"></td>
         </tr>
         <tr>
           <td width='10%'> &nbsp;</td>
           <td width="25%" bgcolor="#4682b4">
             <p align="center">
             <font color="white" face=verdana size=2%>
             <b>Organization</b>
             </font>
           </td>
           <td>
             <input type="radio" name="o" value="NCEAS" checked>NCEAS
             <input type="radio" name="o" value="LTER">LTER
             <input type="radio" name="o" value="NRS">NRS
             <br>
             <input type="radio" name="o" value="PISCO">PISCO
             <input type="radio" name="o" value="OBFS">OBFS
             <input type="radio" name="o" value="Unaffiliated">Unaffiliated
         </tr>
         <tr>
           <td width='10%'> &nbsp;</td>
           <td bgcolor="#4682b4">
             <p align="center">
             <font color="white" face=verdana size=2%>
             <b>Password</b>
             </font>
           </td>
           <td><p><input type="password" name="passwd" value="*******" maxlength="60" size="28">
           </td>
         </tr>
         <tr>
           <td colspan=3 align=center >&nbsp;</td>
         </tr>
       </table>
-            costa
+  In some cases, a Site Contact may need to login to an anonymous account
   rather than his or her personal account. For example, a LTER Information
   Manager may need to login to a dedicated account, named with a three-letter
-            costa
+  acronym, that has been set up for the LTER site. The username
   "GCE" would be used by the LTER Information Mangager at the GCE (Georgia
   Coastal Ecosystems) site.
-            costa
+  </p>
     </li>
     <li>Registering with Harvester
   <p>
   After logging in, you will be presented with a web form that prompts you
   to enter information about your site and how often you want to schedule
   harvests at your site. For example:
-            costa
+      <table bgcolor="#ffffff" border="0" cellpadding="2" width='100%' >
         <tr >
           <td colspan=3 align=center >&nbsp;</td>
         </tr>
         <tr >
           <td colspan=3 align=center >
             <font face=verdana size=1%>
               <b>Metacat Harvester Registration </b>
             </font>
           </td>
         </tr>
         <tr>
           <td width='10%'> &nbsp;</td>
           <td width="25%" bgcolor="#4682b4">
             <p align="center">
             <font color="white" face=verdana size=2%>
             <b>Email address:</b>
             </font>
           </td>
           <td><p><input type="text" size="55" name="uid" value="myname@institution.edu" maxlength="100" size="28"></td>
         </tr>
         <tr>
           <td width='10%'> &nbsp;</td>
           <td bgcolor="#4682b4">
             <p align="center">
             <font color="white" face=verdana size=2%>
             <b>Harvest List URL:</b>
             </font>
           </td>
           <td><p><input type="text" size="55" name="passwd" value="http://somehost.institution.edu/~myname/harvestList.xml" maxlength="60" size="28">
           </td>
         </tr>
         <tr>
           <td colspan=3 align=center >&nbsp;</td>
         </tr>
         <tr>
           <td width='10%'> &nbsp;</td>
           <td bgcolor="#4682b4">
             <p align="center">
             <font color="white" face=verdana size=2%>
             <b>Harvest Frequency (1-99):</b>
             </font>
           </td>
           <td><p><input type="text" size="3" name="passwd" value="2" maxlength="60" size="28">
           </td>
         </tr>
         <tr>
           <td colspan=3 align=center >&nbsp;</td>
         </tr>
         <tr>
           <td width='10%'> &nbsp;</td>
           <td width="25%" bgcolor="#4682b4">
             <p align="center">
             <font color="white" face=verdana size=2%>
             <b>Unit:</b>
             </font>
           </td>
           <td>
             <input type="radio" name="o" value="days" >day(s)
             <input type="radio" name="o" value="weeks" checked>week(s)
             <input type="radio" name="o" value="months">month(s)
         </tr>
       </table>
   <p>
-            costa
+  After values have been entered for each of these fields, click the Register
   button to register your site with Harvester.
   </p>
   <P>
   In the example shown above, Harvester will attempt to harvest documents from
   the site once every 2 weeks, it will access the site's Harvest List at URL
   "http://somehost.institution.edu/~myname/harvestList.xml", and it will send
   email reports to the Site Contact at email address "myname@institution.edu".
   </P>
     </li>
     <li>Unregistering with Harvester
   <p>
   At any time after you have registered with Harvester, you may discontinue
   harvests at your site by unregistering. Simply login as described above and
   then click the Unregister button. After doing so, Harvester will discontinue
   harvests at the site.
   </p>
     </li>
   </ol>
   <h5><a name="Composing">Composing a Harvest List</a></h5>
   <p>
   A Harvest List is an XML file that holds a list of EML documents to be
   harvested. For each EML document in the list, the following information
   must be specified:
   <ul>
     <li><code>docid</code>, which consists of the:
       <ul>
         <li><code>scope</code>, e.g. "demoDocument". The scope is an identifier
             that indicates which group of documents this document belongs to.
         </li>
         <li><code>identifier</code>, e.g. "1". The identifier is a number that
             uniquely identifies this document within the scope.
         </li>
         <li><code>revision</code>, e.g. "5". The revision is a number that
             indicates the current revision of this document.
         </li>
       </ul>
     </li>
     <li><code>documentType</code>, e.g. "eml://ecoinformatics.org/eml-2.0.0".
         The documentType identifies the document as an EML document.</li>
     <li><code>documentURL</code>, e.g. "http://www.lternet.edu/~dcosta/document1.xml".
         The documentURL specifies a place where Harvester can locate
         and retrieve the document via HTTP.</li>
   </ul>
   </p>
   <p>
   The contents of a Harvest List XML file must conform to a particular
   XML Schema, as defined in file <a href="../../lib/harvester/harvestList.xsd">
   harvestList.xsd</a>. The contents of a valid Harvest List
   can best be illustrated by example. The sample Harvest List
   below contains two &lt;<code>document</code>&gt; elements that specify the
   information that Harvester needs to retrieve a pair of EML documents and
   upload them to Metacat:
   <pre>
 &lt;?xml version="1.0" encoding="UTF-8" ?&gt;
 &lt;hrv:harvestList xmlns:hrv="eml://ecoinformatics.org/harvestList" &gt;
     &lt;document&gt;
         &lt;docid&gt;
             &lt;scope&gt;demoDocument&lt;/scope&gt;
             &lt;identifier&gt;1&lt;/identifier&gt;
             &lt;revision&gt;5&lt;/revision&gt;
         &lt;/docid&gt;
         &lt;documentType&gt;eml://ecoinformatics.org/eml-2.0.0&lt;/documentType&gt;
         &lt;documentURL&gt;http://www.lternet.edu/~dcosta/document1.xml&lt;/documentURL&gt;
     &lt;/document&gt;
     &lt;document&gt;
         &lt;docid&gt;
             &lt;scope&gt;demoDocument&lt;/scope&gt;
             &lt;identifier&gt;2&lt;/identifier&gt;
             &lt;revision&gt;1&lt;/revision&gt;
         &lt;/docid&gt;
         &lt;documentType&gt;eml://ecoinformatics.org/eml-2.0.0&lt;/documentType&gt;
         &lt;documentURL&gt;http://www.lternet.edu/~dcosta/document2.xml&lt;/documentURL&gt;
     &lt;/document&gt;
 &lt;/hrv:harvestList&gt;
   </pre>
   <p>
   After editing the Harvest List, ensure that the Harvest List XML file resides
   at the appropriate location on disk as specified by the URL that was entered
   during the <a href=#Registering>registration</a> process.
   </p>
-            costa
+  <p>
   The <a href=./harvestListEditor.html>Harvest List Editor</a> is a tool that
   assists in composing and editing a Harvest List. (Click
   <a href=./harvestListEditor.html>here</a> for additional details.)
   </p>
-            costa
+    <h5><a name="Preparing">Preparing EML Documents for harvest</a></h5>
   <p>
   To prepare a set of EML documents for harvest, ensure that the following is
   true for each document:
   <ul>
     <li>The document contains valid EML</li>
     <li>The document is specified in a &lt;document&gt; element in the
         site's Harvest List, as described above</li>
     <li>The file resides at the appropriate location on disk as specified
         by its URL in the Harvest List</li>
   </ul>
   </p>
     <h5><a name="Reviewing" >Reviewing Harvester Reports to the Site Contact</a></h5>
   <P>
   After every scheduled harvest that takes place at a particular Harvest
   Site, Harvester will send an email report to the Site Contact detailing the
   operations that were performed during the harvest.
   The report will contain information about the operations that were
   performed by Harvester at that site, such as
   which EML documents were harvested and whether any errors were encountered.
   </P>
   <P>
   The Site Contact should review the report, paying particularly
   close attention to any errors that are reported. Errors are indicated
   by operations that display a status value of 1, while operations that
   display a status value of 0 indicate that the operation completed
   successfully.
   </P>
   <p>
   When errors are reported,
   the Site Contact should try to determine whether the source of the error
   is something that can be corrected at the site. Common causes of errors
   might be:
   <ul>
     <li>A document URL specified in the Harvest List does not match
         the location of the actual EML file on the disk</li>
     <li>The Harvest List does not contain valid XML as specified in
         the <a href=../../lib/harvester/harvestList.xsd>harvestList.xsd</a> schema</li>
     <li>The URL to the Harvest List that was specified during
         registration with Harvester does not match the actual location of
         the Harvest List on the disk</li>
     <li>An EML document that Harvester attempted to upload to Metacat does
         not contain valid EML</li>
   </ul>
   </P>
   <p>
   If the Site Contact is unable to determine the cause of the error and its
   resolution, he or she should contact the Harvester Administrator for assistance.
   </p>
   <a href="./properties.html">Back</a> |
   <a href="./metacattour.html">Home</a> |
   <a href="./unimplem.html">Next</a>
 </BODY>
 </HTML>

Project

General

Profile

Metacat