Index: docs/user/metacattour.html
===================================================================
--- docs/user/metacattour.html	(revision 2130)
+++ docs/user/metacattour.html	(revision 2131)
@@ -73,6 +73,7 @@
       <li> <a href="./replication.html">Replication</a></li>
       <li> <a href="./datafiles.html">Non-XML Data Files</a></li>
       <li> <a href="./properties.html">Metacat Properties File</a></li>
+      <li> <a href="./harvester.html">Metacat Harvester</a></li>
     </ul>
   </td>
 </tr>
Index: docs/user/properties.html
===================================================================
--- docs/user/properties.html	(revision 2130)
+++ docs/user/properties.html	(revision 2131)
@@ -22,7 +22,7 @@
       <td class="tablehead" colspan="2"><p class="label">Metacat Properties File</p></td>
       <td class="tablehead" colspan="2" align="right">
         <a href="./datafiles.html">Back</a> | <a href="./metacattour.html">Home</a> | 
-        <a href="./unimplem.html">Next</a>
+        <a href="./harvester.html">Next</a>
       </td>
     </tr>
   </table>
@@ -212,7 +212,7 @@
   
   <br>
   <a href="./datafiles.html">Back</a> | <a href="./metacattour.html">Home</a> | 
-  <a href="./unimplem.html">Next</a>
+  <a href="./harvester.html">Next</a>
   
 
 </BODY>
Index: docs/user/unimplem.html
===================================================================
--- docs/user/unimplem.html	(revision 2130)
+++ docs/user/unimplem.html	(revision 2131)
@@ -21,7 +21,7 @@
     <tr>
       <td class="tablehead" colspan="2"><p class="label">Wishlist</p></td>
       <td class="tablehead" colspan="2" align="right">
-       <a href="./properties.html">Back</a> | <a href="./metacattour.html">Home</a>
+       <a href="./harvester.html">Back</a> | <a href="./metacattour.html">Home</a>
       </td>
     </tr>
   </table>
@@ -33,7 +33,7 @@
   </UL>
   
   <br>
-  <a href="./properties.html">Back</a> | <a href="./metacattour.html">Home</a> 
+  <a href="./harvester.html">Back</a> | <a href="./metacattour.html">Home</a> 
   
 
 </BODY>
Index: docs/user/harvester.html
===================================================================
--- docs/user/harvester.html	(nonexistent)
+++ docs/user/harvester.html	(revision 2131)
@@ -0,0 +1,534 @@
+<!--
+  * harvester.html
+  *
+  *      Authors: Duane Costa
+  *    Copyright: 2004 Regents of the University of California and the
+  *               National Center for Ecological Analysis and Synthesis,
+  *               and the University of New Mexico.
+  *  For Details: http://www.nceas.ucsb.edu/
+  *      Created: 2004 April 9
+  *      Version: 
+  *    File Info: '$ '
+  * 
+  * 
+-->
+<HTML>
+<HEAD>
+<TITLE>Metacat Harvester</TITLE>
+<link rel="stylesheet" type="text/css" href="@docrooturl@default.css">
+</HEAD> 
+<BODY>
+  <table width="100%">
+    <tr>
+      <td class="tablehead" colspan="2">
+        <p class="label">Metacat Harvester</p>
+      </td>
+      <td class="tablehead" colspan="2" align="right">
+        <a href="./properties.html">Back</a> | 
+        <a href="./metacattour.html">Home</a> | 
+        <a href="./unimplem.html">Next</a>
+      </td>
+    </tr>
+  </table>
+  <h4>Introduction</h4>
+The Metacat Harvester (henceforth referred to as "Harvester") is a
+program that automates the retrieval of EML documents from one or more sites
+and their subsequent upload (insert or update) to Metacat. Harvester uses pull
+technology to retrieve and upload documents to Metacat on a regularly
+scheduled basis.
+<P>
+Although Harvester is included with a Metacat installation (beginning with
+Metacat version 1.4.0), it is an extention to Metacat's functionality
+that may be used optionally.
+</P>
+  <h4>Definitions</h4>
+The following table defines a number of terms that are useful in discussing
+Harvester and its features.
+  <br><br>
+  <table border="1">
+    <tr>
+      <td><b>Term</b></td>
+      <td><b>Definition</b></td>
+    </tr>
+    <tr>
+      <td>Harvester</td>
+      <td>The Harvester program, a Java application that is bundled with the
+          Metacat distribution. When a user installs Metacat on a system, 
+          the Harvester program is automatically included in the 
+          installation.
+      </td>
+    </tr>
+    <tr>
+      <td>Harvester Administrator</td>
+      <td>The individual who installs and manages Harvester. Typically, this
+          would be the same individual who installs and manages Metacat at a
+          given installation.
+      </td>
+    </tr>
+    <tr>
+      <td>Harvest Site</td>
+      <td>A location from which Harvester can retrieve EML documents. A given 
+          Harvester can retrieve documents from any number of different 
+          Harvest Sites.
+      </td>
+    </tr>
+    <tr>
+      <td>Harvest</td>
+      <td>The act (by Harvester) of visiting a Harvest Site, retrieving a
+          number of EML documents, and inserting or updating the documents to 
+          Metacat.
+      </td>
+    </tr>
+    <tr>
+      <td>Harvest List</td>
+      <td>An XML document that lists a set of EML documents to be harvested. The
+          Harvest List must conform to an XML Schema, 
+          <a href="../../lib/harvester/harvestList.xsd">harvestList.xsd</a>.
+      </td>
+    </tr>
+    <tr>
+      <td>Site Contact</td>
+      <td>The individual at a particular Harvest Site who registers with 
+          Harvester, composes a Harvest List, and periodically prepares
+          the site's EML documents for retrieval and upload to Metacat.
+      </td>
+    </tr>
+    <tr>
+      <td>Harvest List URL</td>
+      <td>A URL to the Harvest List, as specified by the Site Contact. 
+          Each Harvest Site corresponds to a Harvest List URL. Harvester 
+          uses the URL to locate and read a site's Harvest List.
+      </td>
+    </tr>
+    <tr>
+      <td>Document URL</td>
+      <td>A URL to an EML document, as specified in the Harvest List.
+          The Harvest List may contain any number of Document URLs. Each
+          Document URL provides a locator to a document to be harvested.
+      </td>
+    </tr>
+    <tr>
+      <td>Harvester Registration Page</td>
+      <td>A web page that provides a means for a Site Contact
+          to register with Harvester to schedule regular harvests from the
+          site. Registration involves logging in and then specifying various
+          settings for the Harvest Site, such as the Harvest List URL, the 
+          harvest frequency, and the email address of the Site Contact.
+      </td>
+    </tr>
+  </table>
+  <h4>Managing Harvester</h4>
+  Harvester is managed by the Harvester Administrator. Typically, the same
+  individual who manages a Metacat server would also act as the Harvester
+  Administrator. The responsibilities of the Harvester Administrator include:
+    <ul>
+      <li><a href="#Configuring Harvester">Configuring Harvester</a></li>
+      <li><a href="#Running Harvester">Running Harvester</a></li>
+      <li><a href="#Reviewing Harvester">Reviewing Harvester reports to 
+      the Harvester Administrator</a></li>
+    </ul>
+  <h5><a name="Configuring Harvester">Configuring Harvester</a></h5>
+  <p>Harvester must be configured to interact with a working Metacat
+     installation. Thus, a Metacat installation that has been properly
+     configured and installed is a pre-requisite to running Harvester.
+     Additionally, Harvester has a number of settable properties that
+     control its behavior. All Harvester configuration information is managed 
+     in a single file, 
+     <a href=../../lib/harvester/harvester.properties>harvester.properties</a>, 
+     located at:
+  <pre>      METACAT_HOME/lib/harvester/harvester.properties</pre>
+     where METACAT_HOME is the top-level directory that Metacat is 
+     installed in.
+  </p>
+  <p>The Harvester Administrator should edit 
+     <a href=../../lib/harvester/harvester.properties>harvester.properties</a>, 
+     setting appropriate values for the Metacat URL, database driver,
+     database connection, and other settings. The 
+     following table is a summary of each property and its function.
+  </p>
+  <table border="1">
+    <tr>
+      <td><b>Property</b></td>
+      <td><b>Description</b></td>
+      <td><b>Possible or default value</b></td>
+    </tr>
+    <tr>
+      <td>connectToMetacat</td>
+      <td>This property determines whether Harvester should connect to
+          Metacat to upload documents. It should be set to <code>true</code>
+          under most circumstances. Setting this property to <code>false</code>
+          can be useful for testing whether Harvester is able to retrieve 
+          documents from a site without actually connecting to Metacat to 
+          upload the documents.</td>
+      <td><code>true</code> | <code>false</code><br>
+          Default: <code>true</code>
+    </tr>
+    <tr>
+      <td>dbDriver</td>
+      <td>The JDBC driver to be used to access the backend database. This
+          setting should match the value of the dbDriver property as set 
+          in the <a href=../../build.xml>build.xml</a> file as appropriate 
+          to the database being used (Oracle, PostgreSQL, or SQL Server).
+      </td>
+      <td>Examples:<br>
+          <code>oracle.jdbc.driver.OracleDriver</code><br>
+          <code>org.postgresql.Driver</code><br>
+          <code>com.microsoft.jdbc.sqlserver.SQLServerDriver</code>
+      </td>
+    </tr>
+    <tr>
+      <td>defaultDB</td>
+      <td>The JDBC connection string that Metacat uses to connect to the 
+          backend database. This setting should match the value of
+          the <code>jdbc-connect</code> property as set in the
+          <a href=../../build.properties>build.properties</a>
+          file in the associated Metacat installation.</td>
+      <td>Example:<br>
+          <code>jdbc:oracle:thin:@server.domain.com:1521:Metacat</code></td>
+    </tr>
+    <tr>
+      <td>delay</td>
+      <td>The number of hours that Harvester will wait before beginning its
+          first harvest. For example, if Harvester is run at  1:00 p.m., and
+          the delay is set to 12, Harvester will begin its first harvest at 
+          1:00 a.m.</td>
+      <td>Default: 0</td>
+    </tr>
+    <tr>
+      <td>harvesterAdministrator</td>
+      <td>The email address of the Harvester Administrator. Harvester will
+          send email reports to this address after every harvest.
+      </td>
+      <td>An email address</td>
+    </tr>
+    <tr>
+      <td>logPeriod</td>
+      <td>The number of days that Harvester should retain log entries of harvest
+          operations in the database. Harvester log entries record information
+          such as which documents were harvested, from which sites, and
+          whether any errors were encountered during the harvest. Log entries
+          older than <code>logPeriod</code> number of days are purged from the 
+          database at the end of each harvest.</td>
+      <td>Default: 90</td>
+    </tr>
+    <tr>
+      <td>maxHarvests</td>
+      <td>The maximum number of harvests that Harvester should execute before
+          shutting down. When the Harvester program is executed, it will
+          continue running until it has executed <code>maxHarvests</code>
+          number of harvests and then the program will terminate.</td>
+      <td>Default: 30</td>
+    </tr>
+    <tr>
+      <td>metacatURL</td>
+      <td>The URL of the Metacat servlet to which Harvester should connect
+          for uploading documents.</td>
+      <td>Example:<br>
+               http://somehost.institution.edu:8080/knb/servlet/metacat</td>
+    </tr>
+    <tr>
+      <td>password</td>
+      <td>The password that Harvester uses to access the backend database.
+          This setting should match the value of the <code>password</code>
+          property as set in the 
+          <a href=../../build.properties>build.properties</a>
+          file in the associated Metacat installation.
+      </td>
+      <td>&nbsp;</td>
+    </tr>
+    <tr>
+      <td>period</td>
+      <td>The number of hours between harvests. Harvester will run a new
+          harvest every <code>period</code> number of hours, until the 
+          <code>maxHarvests</code> number of harvests have been run.</td>
+      <td>Default: 24</td>
+    </tr>
+    <tr>
+      <td>smtpServer</td>
+      <td>The SMTP server that Harvester uses for sending email messages
+          to the Harvester Administrator and to Site Contacts.</td>
+      <td>A host name, for example: <code>somehost.institution.edu</code>
+          <br><br>
+          Default: <code>localhost</code>
+          <br><br>
+          Note that the default value will only work if the Harvester 
+          host machine has been configured as a SMTP server.
+      </td>
+    </tr>
+    <tr>
+      <td>user</td>
+      <td>The username that Metacat uses to access the backend database.
+          This setting should match the <code>user</code> value as set in the 
+          <a href=../../build.properties>build.properties</a>
+          file in the associated Metacat installation.
+      </td>
+      <td>&nbsp;</td>
+    </tr>
+    <tr>
+      <td>Harvester Operation Properties (GetDocError, GetDocSuccess, etc.)</td>
+      <td>This group of properties is used by Harvester to report information
+          about the operations it performs for inclusion in log
+          entries and email messages. Under most circumstances the values 
+          of these properties should not be modified.</td>
+      <td>&nbsp;</td>
+    </tr>
+  </table>
+  <br>
+  <h5><a name="Running Harvester">Running Harvester</a></h5>
+  After Harvester has been appropriately 
+  <a href="#Configuring Harvester">configured</a>, 
+  it can be run as follows:
+  <ol>
+  <li>Open a system command window or terminal window.</li>
+  <li>Set the METACAT_HOME environment variable to the value of the Metacat
+      installation directory. Some examples follow:
+      <ul>
+        <li>On Windows:
+        <pre>set METACAT_HOME=C:\somePath\metacat</pre></li>
+        <li>On Linux/Unix (bash shell):
+        <pre>export METACAT_HOME=/home/somePath/metacat</pre></li>
+      </ul>
+  <li>cd to the following directory:
+      <ul>
+        <li>On Windows:
+        <pre>cd %METACAT_HOME%\lib\harvester</pre></li>
+        <li>On Linux/Unix:
+        <pre>cd $METACAT_HOME/lib/harvester</pre></li>
+      </ul>
+  <li>Run the appropriate Harvester shell script, as determined by the
+      operating system:
+      <ul>
+        <li>On Windows:
+        <pre>runHarvester.bat</pre></li>
+        <li>On Linux/Unix:
+        <pre>sh runHarvester.sh</pre></li>
+      </ul>
+  </li>
+  </ol>
+  <p>The Harvester application will start executing. It will begin its first
+  harvest after <code><b>delay</b></code> number of hours (as specified in the
+  <a href=../../lib/harvester/harvester.properties>harvester.properties</a>
+  file). The application will continue running a new harvest every
+  <code><b>period</b></code> number of hours until a <code><b>maxHarvests</b></code>
+  number of harvests have been completed.
+  </p>
+  <h5><a name="Reviewing Harvester">
+  Reviewing Harvester Reports to the Harvester Administrator</a></h5>
+  <P>
+  After every harvest, Harvester will send an email report to the Harvester
+  Administrator detailing the operations that were performed during the
+  harvest. The report will contain information about each of the Harvest Sites
+  that were harvested from, such as which EML documents were
+  harvested and whether any errors were encountered.
+  </P>
+  <p>
+  The harvest report will contain a list of log entries, where each log entry
+  describes an operation that was performed by Harvester. Log entries that
+  show a status value of 1 indicate that an error occurred during the
+  operation, while those that show a status value of 0 indicate that the
+  operation was completed successfully.
+  </p>
+  <P>The Harvester Administrator should review the report, paying particularly 
+  close attention to any errors that are reported and to the accompanying error
+  messages that are displayed. When errors are reported at
+  a particular site, the Harvester Administrator should contact the Site
+  Contact to determine the source of the error and its resolution. See 
+  <a href=#Reviewing>Reviewing Harvester Reports to the Site Contact</a> for a
+  description of common sources of errors at a Harvest Site.
+  </P>
+  <p>Errors that are independent of a particular site may indicate a problem 
+  with Harvester itself, Metacat, or the database connection. Refer to the
+  error message to determine the source of the error and its resolution.
+  </p>
+  <h4>Managing a Harvest Site</h4>
+  A Harvest Site is managed by a Site Contact.
+  The responsibilities of a Site Contact fall into the following categories:
+    <ul>
+      <li><a href=#Registering>Registering with Harvester</a></li>
+      <li><a href=#Composing>Composing a Harvest List</a></li>
+      <li><a href=#Preparing>Preparing EML Documents for harvest</a></li>
+      <li><a href=#Reviewing>Reviewing Harvester reports to the Site Contact</a></li>
+    </ul>
+    <h5><a name="Registering">Registering with Harvester</a></h5>
+  <p>
+  A Site Contact registers a site with Harvester by logging in to the
+  Harvester Registration page and entering several items of information
+  that Harvester needs to know about the site.
+  </p>
+  <ol>
+    <li>Logging in to the Harvester Registration Page
+  <p>
+  The Harvester Registration page is accessed from Metacat. For example, if
+  the Metacat server that you wish to register with resides at the following 
+  URL:
+  <pre>  http://somehost.somelocation.edu:8080/knb/index.jsp</pre>
+  then the Harvester Registration page would be accessed at:
+  <pre>  http://somehost.somelocation.edu:8080/knb/style/skins/dev/harvesterRegistrationLogin.html</pre>
+  </p>
+  <p>
+  After bringing up this page in your browser, login to your Metacat account 
+  by entering your username and password.
+  The username should include the full LDAP specification, for example:
+  <pre>
+  Username:   uid=jdoe,o=lter,dc=ecoinformatics,dc=org
+  Password:   *******
+  </pre>
+  In some cases, a Site Contact may need to login to an anonymous account
+  rather than his or her personal account. For example, a LTER Information 
+  Manager may need to login to a dedicated account, named with a three-letter 
+  acronym, that has been set up for the LTER site. For example:
+  <pre>
+  Username:   uid=GCE,o=lter,dc=ecoinformatics,dc=org
+  Password:   *******
+  </pre>
+  is the account login that would be used by the LTER Information Mangager
+  at the GCE (Georgia Coastal Ecosystems) site.
+  </p>
+    </li>
+    <li>Registering with Harvester
+  <p>
+  After logging in, you will be presented with a web form that prompts you
+  to enter information about your site and how often you want to schedule
+  harvests at your site. For example:
+  </p>
+  <pre>
+  Email address:            myname@institution.edu
+  Harvest List URL:         http://somehost.institution.edu/~myname/harvestList.xml
+  Harvest Frequency (1-99): 2
+  Unit:                     ( ) day(s)    (*) week(s)   ( ) month(s)
+  </pre>
+  After values have been entered for each of these fields, click the Register 
+  button to register your site with Harvester.
+  </p>
+  <P>
+  In the example shown above, Harvester will attempt to harvest documents from 
+  the site once every 2 weeks, it will access the site's Harvest List at URL
+  "http://somehost.institution.edu/~myname/harvestList.xml", and it will send
+  email reports to the Site Contact at email address "myname@institution.edu".
+  </P>
+    </li>
+    <li>Unregistering with Harvester
+  <p>
+  At any time after you have registered with Harvester, you may discontinue 
+  harvests at your site by unregistering. Simply login as described above and 
+  then click the Unregister button. After doing so, Harvester will discontinue 
+  harvests at the site.
+  </p>
+    </li>
+  </ol>
+  <h5><a name="Composing">Composing a Harvest List</a></h5>
+  <p>
+  A Harvest List is an XML file that holds a list of EML documents to be
+  harvested. For each EML document in the list, the following information
+  must be specified:
+  <ul>
+    <li><code>docid</code>, which consists of the:
+      <ul>
+        <li><code>scope</code>, e.g. "demoDocument". The scope is an identifier 
+            that indicates which group of documents this document belongs to.
+        </li>
+        <li><code>identifier</code>, e.g. "1". The identifier is a number that 
+            uniquely identifies this document within the scope.
+        </li>
+        <li><code>revision</code>, e.g. "5". The revision is a number that 
+            indicates the current revision of this document.
+        </li>
+      </ul>
+    </li>
+    <li><code>documentType</code>, e.g. "eml://ecoinformatics.org/eml-2.0.0".
+        The documentType identifies the document as an EML document.</li>
+    <li><code>documentURL</code>, e.g. "http://www.lternet.edu/~dcosta/document1.xml".
+        The documentURL specifies a place where Harvester can locate 
+        and retrieve the document via HTTP.</li>
+  </ul>
+  </p>
+  <p>
+  The contents of a Harvest List XML file must conform to a particular
+  XML Schema, as defined in file <a href="../../lib/harvester/harvestList.xsd">
+  harvestList.xsd</a>. The contents of a valid Harvest List 
+  can best be illustrated by example. The sample Harvest List
+  below contains two &lt;<code>document</code>&gt; elements that specify the 
+  information that Harvester needs to retrieve a pair of EML documents and 
+  upload them to Metacat:
+  <pre>
+&lt;?xml version="1.0" encoding="UTF-8" ?&gt;
+&lt;hrv:harvestList xmlns:hrv="eml://ecoinformatics.org/harvestList" &gt;
+    &lt;document&gt;
+        &lt;docid&gt;
+            &lt;scope&gt;demoDocument&lt;/scope&gt;
+            &lt;identifier&gt;1&lt;/identifier&gt;
+            &lt;revision&gt;5&lt;/revision&gt;
+        &lt;/docid&gt;
+        &lt;documentType&gt;eml://ecoinformatics.org/eml-2.0.0&lt;/documentType&gt;
+        &lt;documentURL&gt;http://www.lternet.edu/~dcosta/document1.xml&lt;/documentURL&gt;
+    &lt;/document&gt;
+    &lt;document&gt;
+        &lt;docid&gt;
+            &lt;scope&gt;demoDocument&lt;/scope&gt;
+            &lt;identifier&gt;2&lt;/identifier&gt;
+            &lt;revision&gt;1&lt;/revision&gt;
+        &lt;/docid&gt;
+        &lt;documentType&gt;eml://ecoinformatics.org/eml-2.0.0&lt;/documentType&gt;
+        &lt;documentURL&gt;http://www.lternet.edu/~dcosta/document2.xml&lt;/documentURL&gt;
+    &lt;/document&gt;
+&lt;/hrv:harvestList&gt;
+  </pre>
+  <p>
+  After editing the Harvest List, ensure that the Harvest List XML file resides
+  at the appropriate location on disk as specified by the URL that was entered
+  during the <a href=#Registering>registration</a> process.
+  </p>
+    <h5><a name="Preparing">Preparing EML Documents for harvest</a></h5>
+  <p>
+  To prepare a set of EML documents for harvest, ensure that the following is 
+  true for each document:
+  <ul>
+    <li>The document contains valid EML</li>
+    <li>The document is specified in a &lt;document&gt; element in the 
+        site's Harvest List, as described above</li>
+    <li>The file resides at the appropriate location on disk as specified
+        by its URL in the Harvest List</li>
+  </ul>
+  </p>
+    <h5><a name="Reviewing" >Reviewing Harvester Reports to the Site Contact</a></h5>
+  <P>
+  After every scheduled harvest that takes place at a particular Harvest
+  Site, Harvester will send an email report to the Site Contact detailing the 
+  operations that were performed during the harvest.
+  The report will contain information about the operations that were
+  performed by Harvester at that site, such as 
+  which EML documents were harvested and whether any errors were encountered.
+  </P>
+  <P>
+  The Site Contact should review the report, paying particularly 
+  close attention to any errors that are reported. Errors are indicated
+  by operations that display a status value of 1, while operations that
+  display a status value of 0 indicate that the operation completed
+  successfully.
+  </P>
+  <p>
+  When errors are reported,
+  the Site Contact should try to determine whether the source of the error
+  is something that can be corrected at the site. Common causes of errors 
+  might be:
+  <ul>
+    <li>A document URL specified in the Harvest List does not match
+        the location of the actual EML file on the disk</li>
+    <li>The Harvest List does not contain valid XML as specified in
+        the <a href=../../lib/harvester/harvestList.xsd>harvestList.xsd</a> schema</li>
+    <li>The URL to the Harvest List that was specified during
+        registration with Harvester does not match the actual location of
+        the Harvest List on the disk</li>
+    <li>An EML document that Harvester attempted to upload to Metacat does
+        not contain valid EML</li>
+  </ul>
+  </P> 
+  <p>
+  If the Site Contact is unable to determine the cause of the error and its
+  resolution, he or she should contact the Harvester Administrator for assistance.
+  </p>
+  <a href="./properties.html">Back</a> | 
+  <a href="./metacattour.html">Home</a> | 
+  <a href="./unimplem.html">Next</a>
+</BODY>
+</HTML>

Property changes on: docs/user/harvester.html
___________________________________________________________________
Added: svn:eol-style
## -0,0 +1 ##
+native
\ No newline at end of property
Added: svn:executable
## -0,0 +1 ##
+*
\ No newline at end of property
Added: svn:keywords
## -0,0 +1 ##
+Author Date Id Revision
\ No newline at end of property
