Metacat

<br/><br/>

<hr/><hr/>

<br/><br/>

  Harvester is managed by the Harvester Administrator. Typically, the same

  individual who manages a Metacat server would also act as the Harvester

  Administrator. The responsibilities of the Harvester Administrator include:

    <ul>

      <li><a href="#Configuring Harvester">Configuring Harvester</a></li>

      <li><a href="#Running Harvester">Running Harvester</a></li>

      <li><a href="#Reviewing Harvester">Reviewing Harvester reports to 

      the Harvester Administrator</a></li>

    </ul>

  <h5><a name="Configuring Harvester">Configuring Harvester</a></h5>

  <p>Harvester must be configured to interact with a working Metacat

     installation. Thus, a Metacat installation that has been properly

     configured and installed is a pre-requisite to running Harvester.

     Additionally, Harvester has a number of settable properties that

     control its behavior. All Harvester configuration information is managed 

     in a single file, 

     <a href=../../lib/metacat.properties>metacat.properties</a>, 

     located at:

  <pre>      METACAT_HOME/lib/metacat.properties</pre>

     where METACAT_HOME is the top-level directory that Metacat is 

     installed in. 

  </p>

  <p>Harvester properties are grouped together in 

     <a href=../../lib/metacat.properties>metacat.properties</a>, beginning

     after the comment line:

  <pre><code>      # Harvester properties</code></pre>

  </p>

  <p>The Harvester Administrator should edit 

     <a href=../../lib/metacat.properties>metacat.properties</a>, 

     setting appropriate values for the <code><b>harvesterAdministrator</b></code>

     property, the <code><b>smtpServer</b></code> property, and possibly other 

     properties. The following table is a summary of each property and its function.

  </p>

  <table border="1">

    <tr>

      <td><b>Property</b></td>

      <td><b>Description</b></td>

      <td><b>Possible or default value</b></td>

    </tr>

    <tr>

      <td>connectToMetacat</td>

      <td>This property determines whether Harvester should connect to

          Metacat to upload documents. It should be set to <code>true</code>

          under most circumstances. Setting this property to <code>false</code>

          can be useful for testing whether Harvester is able to retrieve 

          documents from a site without actually connecting to Metacat to 

          upload the documents.</td>

      <td><code>true</code> | <code>false</code><br>

          Default: <code>true</code>

    </tr>

    <tr>

      <td>delay</td>

      <td>The number of hours that Harvester will wait before beginning its

          first harvest. For example, if Harvester is run at  1:00 p.m., and

          the delay is set to 12, Harvester will begin its first harvest at 

          1:00 a.m.</td>

      <td>Default: 0</td>

    </tr>

    <tr>

      <td>harvesterAdministrator</td>

      <td>The email address of the Harvester Administrator. Harvester will

          send email reports to this address after every harvest. You may

          enter multiple email addresses by separating each address with

          a comma or semicolon, for example, "name1@abc.edu,name2@abc.edu".

      </td>

      <td>An email address, or multiple email addresses separated by commas

          or semi-colons</td>

    </tr>

    <tr>

      <td>logPeriod</td>

      <td>The number of days that Harvester should retain log entries of harvest

          operations in the database. Harvester log entries record information

          such as which documents were harvested, from which sites, and

          whether any errors were encountered during the harvest. Log entries

          older than <code>logPeriod</code> number of days are purged from the 

          database at the end of each harvest.</td>

      <td>Default: 90</td>

    </tr>

    <tr>

      <td>maxHarvests</td>

      <td>The maximum number of harvests that Harvester should execute before

          shutting down. When the Harvester program is executed, it will

          continue running until it has executed <code>maxHarvests</code>

          number of harvests and then the program will terminate. If

          the value of <code>maxHarvests</code> is set to 0 or a negative

          number, it will be ignored and Harvester will execute indefinitely.

          </td>

      <td>Default: 0</td>

    </tr>

    <tr>

      <td>period</td>

      <td>The number of hours between harvests. Harvester will run a new

          harvest every <code>period</code> number of hours, until the 

          <code>maxHarvests</code> number of harvests have been run, or

          indefinitely if <code>maxHarvests</code> is set to a value of

          0 or a negative number.

      <td>Default: 24</td>

    </tr>

    <tr>

      <td>smtpServer</td>

      <td>The SMTP server that Harvester uses for sending email messages

          to the Harvester Administrator and to Site Contacts.</td>

      <td>A host name, for example: <code>somehost.institution.edu</code>

          <br><br>

          Default: <code>localhost</code>

          <br><br>

          Note that the default value will only work if the Harvester 

          host machine has been configured as a SMTP server.

      </td>

    </tr>

    <tr>

      <td>Harvester Operation Properties (GetDocError, GetDocSuccess, etc.)</td>

      <td>This group of properties is used by Harvester to report information

          about the operations it performs for inclusion in log

          entries and email messages. Under most circumstances the values 

          of these properties should not be modified.</td>

      <td>&nbsp;</td>

    </tr>

  </table>

  <br>

  <h5><a name="Running Harvester">Running Harvester</a></h5>

  After Harvester has been appropriately 

  <a href="#Configuring Harvester">configured</a>, 

  it can be run in either of two ways: (A) in a command window, or, (B)

  as a servlet. If you wish only to test that Harvester is functioning, 

  or if you expect to use Harvester infrequently, it may desirable to run it from a 

  command window. However, under most circumstances you will want Harvester to 

  run continuously as a background servlet process. This eliminates the

  need to keep a command window continuously open while Harvester is running.

  Both of these procedures are described below.

  <ul>

  <li> (A) Running Harvester in a Command Window

  <ol>

  <li>Open a system command window or terminal window.</li>

  <li>Set the METACAT_HOME environment variable to the value of the Metacat

      installation directory. Some examples follow:

      <ul>

        <li>On Windows:

        <pre>set METACAT_HOME=C:\somePath\metacat</pre></li>

        <li>On Linux/Unix (bash shell):

        <pre>export METACAT_HOME=/home/somePath/metacat</pre></li>

      </ul>

  <li>cd to the following directory:

      <ul>

        <li>On Windows:

        <pre>cd %METACAT_HOME%\lib\harvester</pre></li>

        <li>On Linux/Unix:

        <pre>cd $METACAT_HOME/lib/harvester</pre></li>

      </ul>

  <li>Run the appropriate Harvester shell script, as determined by the

      operating system:

      <ul>

        <li>On Windows:

        <pre>runHarvester.bat</pre></li>

        <li>On Linux/Unix:

        <pre>sh runHarvester.sh</pre></li>

      </ul>

  </li>

 </ol>

  <p>The Harvester application will start executing. It will begin its first

  harvest after <code><b>delay</b></code> number of hours (as specified in the

  <a href=../../lib/metacat.properties>metacat.properties</a>

  file). The application will continue running a new harvest every

  <code><b>period</b></code> number of hours until a <code><b>maxHarvests</b></code>

  number of harvests have been completed (if <code><b>maxHarvests</b></code> is set 

  to a value greater than 0), or until you interrupt the process by hitting CTRL/C 

  in the command window.

  </p>

  </li>

  <li> (B) Running Harvester as a Servlet

  <ol>

  <li>Edit the file in your Metcat installation, <code>lib/web.xml.<em>tomcatN</em></code>, where <em>tomcatN</em> corresponds to the

  version of Tomcat you are running. For example, if you are running Tomcat 5,

  edit file <code>lib/web.xml.tomcat5</code>.</li>

  <li>Remove the comment symbols around the HarvesterServlet entry, so that:

  <pre><code>

  <!--

  <servlet>

  <servlet-name>HarvesterServlet</servlet-name>

  <servlet-class>edu.ucsb.nceas.metacat.harvesterClient.HarvesterServlet</servlet-class>

  <init-param>

    <param-name>debug</param-name>

    <param-value>1</param-value>

  </init-param>

  <init-param>

    <param-name>listings</param-name>

    <param-value>true</param-value>

  </init-param>

  <load-on-startup>1</load-on-startup>

  </servlet>

  -->

  </code></pre>

  is changed to:

  <pre><code>

  <servlet>

  <servlet-name>HarvesterServlet</servlet-name>

  <servlet-class>edu.ucsb.nceas.metacat.harvesterClient.HarvesterServlet</servlet-class>

  <init-param>

    <param-name>debug</param-name>

    <param-value>1</param-value>

  </init-param>

  <init-param>

    <param-name>listings</param-name>

    <param-value>true</param-value>

  </init-param>

  <load-on-startup>1</load-on-startup>

  </servlet>

  </code></pre>

  Save the edited file.

  </li>

  <li>Shutdown Tomcat.</li>

  <li>Redeploy Metacat by running the following two ant commands from the top-level

  directory of your Metacat installation:

  <code><pre>

  ant cleanweb

  ant install</code></pre>

  </li>

  <li>Restart Tomcat.</li>

 </ol>

  <p>About thirty seconds after you restart Tomcat, the Harvester servlet will 

  start executing. It will begin its first

  harvest after <code><b>delay</b></code> number of hours (as specified in the

  <a href=../../lib/metacat.properties>metacat.properties</a>

  file). The servlet will continue running a new harvest every

  <code><b>period</b></code> number of hours until a <code><b>maxHarvests</b></code>

  number of harvests have been completed (if <code><b>maxHarvests</b></code> is set

  to a value greater than 0), or until Tomcat shuts down.

  </p>

  </li>

   <h5><a name="Reviewing Harvester">

  Reviewing Harvester Reports to the Harvester Administrator</a></h5>

  <P>

  After every harvest, Harvester will send an email report to the Harvester

  Administrator detailing the operations that were performed during the

  harvest. The report will contain information about each of the Harvest Sites

  that were harvested from, such as which EML documents were

  harvested and whether any errors were encountered.

  </P>

  <p>

  The harvest report will contain a list of log entries, where each log entry

  describes an operation that was performed by Harvester. Log entries that

  show a status value of 1 indicate that an error occurred during the

  operation, while those that show a status value of 0 indicate that the

  operation was completed successfully.

  </p>

  <P>The Harvester Administrator should review the report, paying particularly 

  close attention to any errors that are reported and to the accompanying error

  messages that are displayed. When errors are reported at

  a particular site, the Harvester Administrator should contact the Site

  Contact to determine the source of the error and its resolution. See 

  <a href=#Reviewing>Reviewing Harvester Reports to the Site Contact</a> for a

  description of common sources of errors at a Harvest Site.

  </P>

  <p>Errors that are independent of a particular site may indicate a problem 

  with Harvester itself, Metacat, or the database connection. Refer to the

  error message to determine the source of the error and its resolution.

  </p>

  <h4>Managing a Harvest Site</h4>

  A Harvest Site is managed by a Site Contact.

  The responsibilities of a Site Contact fall into the following categories:

    <ul>

      <li><a href=#Registering>Registering with Harvester</a></li>

      <li><a href=#Composing>Composing a Harvest List</a></li>

      <li><a href=#Preparing>Preparing EML Documents for harvest</a></li>

      <li><a href=#Reviewing>Reviewing Harvester reports to the Site Contact</a></li>

    </ul>

    <h5><a name="Registering">Registering with Harvester</a></h5>

  <p>

  A Site Contact registers a site with Harvester by logging in to the

  Harvester Registration page and entering several items of information

  that Harvester needs to know about the site.

  </p>

  <ol>

    <li>Logging in to the Harvester Registration Page

  <p>

  The Harvester Registration page is accessed from Metacat. For example, if

  the Metacat server that you wish to register with resides at the following 

  URL:

  <pre>  http://somehost.somelocation.edu:8080/knb/index.jsp</pre>

  then the Harvester Registration page would be accessed at:

  <pre>  http://somehost.somelocation.edu:8080/knb/style/skins/knb/harvesterRegistrationLogin.html</pre>

  </p>

  <p>

  After bringing up this page in your browser, login to your Metacat account 

  by entering your username, organization, and password. For example:

      <table bgcolor="#ffffff" border="0" cellpadding="2" width='100%' >

        <tr > 

          <td colspan=3 align=center >&nbsp;</td> 

        </tr>

        <tr > 

          <td colspan=3 align=center >

            <font face=verdana size=1%>

              <b>Please  Enter Username, Organization, and Password </b>

            </font>

          </td> 

        </tr>

        <tr>

          <td width='10%'> &nbsp;</td>

          <td width="25%" bgcolor="#4682b4">

            <p align="center">

            <font color="white" face=verdana size=2%>

            <b>Username</b>

            </font>

          </td>

          <td><p><input type="text" name="uid" value="jdoe" maxlength="100" size="28"></td>

        </tr>

        <tr>

          <td width='10%'> &nbsp;</td>

          <td width="25%" bgcolor="#4682b4">

            <p align="center">

            <font color="white" face=verdana size=2%>

            <b>Organization</b>

            </font>

          </td>

          <td>

            <input type="radio" name="o" value="NCEAS" checked>NCEAS

            <input type="radio" name="o" value="LTER">LTER

            <input type="radio" name="o" value="NRS">NRS

            <br>

            <input type="radio" name="o" value="PISCO">PISCO

            <input type="radio" name="o" value="OBFS">OBFS

            <input type="radio" name="o" value="Unaffiliated">Unaffiliated

        </tr>

        <tr>

          <td width='10%'> &nbsp;</td>

          <td bgcolor="#4682b4">

            <p align="center">

            <font color="white" face=verdana size=2%>

            <b>Password</b>

            </font>

          </td>

          <td><p><input type="password" name="passwd" value="*******" maxlength="60" size="28">

          </td>

        </tr>

        <tr> 

          <td colspan=3 align=center >&nbsp;</td>

        </tr>

      </table>

  In some cases, a Site Contact may need to login to an anonymous account

  rather than his or her personal account. For example, a LTER Information 

  Manager may need to login to a dedicated account, named with a three-letter 

  acronym, that has been set up for the LTER site. The username

  "GCE" would be used by the LTER Information Mangager at the GCE (Georgia 

  Coastal Ecosystems) site.

  </p>

    </li>

    <li>Registering with Harvester

  <p>

  After logging in, you will be presented with a web form that prompts you

  to enter information about your site and how often you want to schedule

  harvests at your site. For example:

      <table bgcolor="#ffffff" border="0" cellpadding="2" width='100%' >

        <tr > 

          <td colspan=3 align=center >&nbsp;</td> 

        </tr>

        <tr > 

          <td colspan=3 align=center >

            <font face=verdana size=1%>

              <b>Metacat Harvester Registration </b>

            </font>

          </td> 

        </tr>

        <tr>

          <td width='10%'> &nbsp;</td>

          <td width="25%" bgcolor="#4682b4">

            <p align="center">

            <font color="white" face=verdana size=2%>

            <b>Email address:</b>

            </font>

          </td>

          <td><p><input type="text" size="55" name="uid" value="myname@institution.edu" maxlength="100" size="28"></td>

        </tr>

        <tr>

          <td width='10%'> &nbsp;</td>

          <td bgcolor="#4682b4">

            <p align="center">

            <font color="white" face=verdana size=2%>

            <b>Harvest List URL:</b>

            </font>

          </td>

          <td><p><input type="text" size="55" name="passwd" value="http://somehost.institution.edu/~myname/harvestList.xml" maxlength="60" size="28">

          </td>

        </tr>

        <tr> 

          <td colspan=3 align=center >&nbsp;</td>

        </tr>

        <tr>

          <td width='10%'> &nbsp;</td>

          <td bgcolor="#4682b4">

            <p align="center">

            <font color="white" face=verdana size=2%>

            <b>Harvest Frequency (1-99):</b>

            </font>

          </td>

          <td><p><input type="text" size="3" name="passwd" value="2" maxlength="60" size="28">

          </td>

        </tr>

        <tr> 

          <td colspan=3 align=center >&nbsp;</td>

        </tr>

        <tr>

          <td width='10%'> &nbsp;</td>

          <td width="25%" bgcolor="#4682b4">

            <p align="center">

            <font color="white" face=verdana size=2%>

            <b>Unit:</b>

            </font>

          </td>

          <td>

            <input type="radio" name="o" value="days" >day(s)

            <input type="radio" name="o" value="weeks" checked>week(s)

            <input type="radio" name="o" value="months">month(s)

        </tr>

      </table>

  <p>

  After values have been entered for each of these fields, click the Register 

  button to register your site with Harvester.

  </p>

  <P>

  In the example shown above, Harvester will attempt to harvest documents from 

  the site once every 2 weeks, it will access the site's Harvest List at URL

  "http://somehost.institution.edu/~myname/harvestList.xml", and it will send

  email reports to the Site Contact at email address "myname@institution.edu".

  </P>

  <P>

  Note that you may enter multiple email addresses by separating each

  address with a comma or a semi-colon. For example, 

  "myname@institution.edu,anothername@institution.edu".

  </P>

    </li>

    <li>Unregistering with Harvester

  <p>

  At any time after you have registered with Harvester, you may discontinue 

  harvests at your site by unregistering. Simply login as described above and 

  then click the Unregister button. After doing so, Harvester will discontinue 

  harvests at the site.

  </p>

    </li>

  </ol>

  <h5><a name="Composing">Composing a Harvest List</a></h5>

  <p>

  A Harvest List is an XML file that holds a list of EML documents to be

  harvested. For each EML document in the list, the following information

  must be specified:

  <ul>

    <li><code>docid</code>, which consists of the:

      <ul>

        <li><code>scope</code>, e.g. "demoDocument". The scope is an identifier 

            that indicates which group of documents this document belongs to.

        </li>

        <li><code>identifier</code>, e.g. "1". The identifier is a number that 

            uniquely identifies this document within the scope.

        </li>

        <li><code>revision</code>, e.g. "5". The revision is a number that 

            indicates the current revision of this document.

        </li>

      </ul>

    </li>

    <li><code>documentType</code>, e.g. "eml://ecoinformatics.org/eml-2.0.0".

        The documentType identifies the document as an EML document.</li>

    <li><code>documentURL</code>, e.g. "http://www.lternet.edu/~dcosta/document1.xml".

        The documentURL specifies a place where Harvester can locate 

        and retrieve the document via HTTP.</li>

  </ul>

  </p>

  <p>

  The contents of a Harvest List XML file must conform to a particular

  XML Schema, as defined in file <a href="../../lib/harvester/harvestList.xsd">

  harvestList.xsd</a>. The contents of a valid Harvest List 

  can best be illustrated by example. The sample Harvest List

  below contains two &lt;<code>document</code>&gt; elements that specify the 

  information that Harvester needs to retrieve a pair of EML documents and 

  upload them to Metacat:

  <pre>

<?xml version="1.0" encoding="UTF-8" ?>

<hrv:harvestList xmlns:hrv="eml://ecoinformatics.org/harvestList" >

    <document>

        <docid>

            <scope>demoDocument</scope>

            <identifier>1</identifier>

            <revision>5</revision>

        </docid>

        <documentType>eml://ecoinformatics.org/eml-2.0.0</documentType>

        <documentURL>http://www.lternet.edu/~dcosta/document1.xml</documentURL>

    </document>

    <document>

        <docid>

            <scope>demoDocument</scope>

            <identifier>2</identifier>

            <revision>1</revision>

        </docid>

        <documentType>eml://ecoinformatics.org/eml-2.0.0</documentType>

        <documentURL>http://www.lternet.edu/~dcosta/document2.xml</documentURL>

    </document>

</hrv:harvestList>

  </pre>

  <p>

  After editing the Harvest List, ensure that the Harvest List XML file resides

  at the appropriate location on disk as specified by the URL that was entered

  during the <a href=#Registering>registration</a> process.

  </p>

  <p>

  The <a href=./harvestListEditor.html>Harvest List Editor</a> is a tool that 

  assists in composing and editing a Harvest List. (Click 

  <a href=./harvestListEditor.html>here</a> for additional details.)

  </p>

    <h5><a name="Preparing">Preparing EML Documents for harvest</a></h5>

  <p>

  To prepare a set of EML documents for harvest, ensure that the following is 

  true for each document:

  <ul>

    <li>The document contains valid EML</li>

    <li>The document is specified in a &lt;document&gt; element in the 

        site's Harvest List, as described above</li>

    <li>The file resides at the appropriate location on disk as specified

        by its URL in the Harvest List</li>

  </ul>

  </p>

    <h5><a name="Reviewing" >Reviewing Harvester Reports to the Site Contact</a></h5>

  <P>

  After every scheduled harvest that takes place at a particular Harvest

  Site, Harvester will send an email report to the Site Contact detailing the 

  operations that were performed during the harvest.

  The report will contain information about the operations that were

  performed by Harvester at that site, such as 

  which EML documents were harvested and whether any errors were encountered.

  </P>

  <P>

  The Site Contact should review the report, paying particularly 

  close attention to any errors that are reported. Errors are indicated

  by operations that display a status value of 1, while operations that

  display a status value of 0 indicate that the operation completed

  successfully.

  </P>

  <p>

  When errors are reported,

  the Site Contact should try to determine whether the source of the error

  is something that can be corrected at the site. Common causes of errors 

  might be:

  <ul>

    <li>A document URL specified in the Harvest List does not match

        the location of the actual EML file on the disk</li>

    <li>The Harvest List does not contain valid XML as specified in

        the <a href=../../lib/harvester/harvestList.xsd>harvestList.xsd</a> schema</li>

    <li>The URL to the Harvest List that was specified during

        registration with Harvester does not match the actual location of

        the Harvest List on the disk</li>

    <li>An EML document that Harvester attempted to upload to Metacat does

        not contain valid EML</li>

  </ul>

  </P> 

  <p>

  If the Site Contact is unable to determine the cause of the error and its

  resolution, he or she should contact the Harvester Administrator for assistance.

  </p>

  <a href="./properties.html">Back</a> | 

  <a href="./metacattour.html">Home</a> | 

  <a href="./unimplem.html">Next</a>

       <a href="./harvester.html">Back</a> | <a href="./metacattour.html">Home</a>