|
1 |
<!--
|
|
2 |
* harvester.html
|
|
3 |
*
|
|
4 |
* Authors: Duane Costa
|
|
5 |
* Copyright: 2004 Regents of the University of California and the
|
|
6 |
* National Center for Ecological Analysis and Synthesis,
|
|
7 |
* and the University of New Mexico.
|
|
8 |
* For Details: http://www.nceas.ucsb.edu/
|
|
9 |
* Created: 2004 April 9
|
|
10 |
* Version:
|
|
11 |
* File Info: '$ '
|
|
12 |
*
|
|
13 |
*
|
|
14 |
-->
|
|
15 |
<HTML>
|
|
16 |
<HEAD>
|
|
17 |
<TITLE>Metacat Harvester</TITLE>
|
|
18 |
<link rel="stylesheet" type="text/css" href="@docrooturl@default.css">
|
|
19 |
</HEAD>
|
|
20 |
<BODY>
|
|
21 |
<table width="100%">
|
|
22 |
<tr>
|
|
23 |
<td class="tablehead" colspan="2">
|
|
24 |
<p class="label">Metacat Harvester</p>
|
|
25 |
</td>
|
|
26 |
<td class="tablehead" colspan="2" align="right">
|
|
27 |
<a href="./properties.html">Back</a> |
|
|
28 |
<a href="./metacattour.html">Home</a> |
|
|
29 |
<a href="./unimplem.html">Next</a>
|
|
30 |
</td>
|
|
31 |
</tr>
|
|
32 |
</table>
|
|
33 |
<h4>Introduction</h4>
|
|
34 |
The Metacat Harvester (henceforth referred to as "Harvester") is a
|
|
35 |
program that automates the retrieval of EML documents from one or more sites
|
|
36 |
and their subsequent upload (insert or update) to Metacat. Harvester uses pull
|
|
37 |
technology to retrieve and upload documents to Metacat on a regularly
|
|
38 |
scheduled basis.
|
|
39 |
<P>
|
|
40 |
Although Harvester is included with a Metacat installation (beginning with
|
|
41 |
Metacat version 1.4.0), it is an extention to Metacat's functionality
|
|
42 |
that may be used optionally.
|
|
43 |
</P>
|
|
44 |
<h4>Definitions</h4>
|
|
45 |
The following table defines a number of terms that are useful in discussing
|
|
46 |
Harvester and its features.
|
|
47 |
<br><br>
|
|
48 |
<table border="1">
|
|
49 |
<tr>
|
|
50 |
<td><b>Term</b></td>
|
|
51 |
<td><b>Definition</b></td>
|
|
52 |
</tr>
|
|
53 |
<tr>
|
|
54 |
<td>Harvester</td>
|
|
55 |
<td>The Harvester program, a Java application that is bundled with the
|
|
56 |
Metacat distribution. When a user installs Metacat on a system,
|
|
57 |
the Harvester program is automatically included in the
|
|
58 |
installation.
|
|
59 |
</td>
|
|
60 |
</tr>
|
|
61 |
<tr>
|
|
62 |
<td>Harvester Administrator</td>
|
|
63 |
<td>The individual who installs and manages Harvester. Typically, this
|
|
64 |
would be the same individual who installs and manages Metacat at a
|
|
65 |
given installation.
|
|
66 |
</td>
|
|
67 |
</tr>
|
|
68 |
<tr>
|
|
69 |
<td>Harvest Site</td>
|
|
70 |
<td>A location from which Harvester can retrieve EML documents. A given
|
|
71 |
Harvester can retrieve documents from any number of different
|
|
72 |
Harvest Sites.
|
|
73 |
</td>
|
|
74 |
</tr>
|
|
75 |
<tr>
|
|
76 |
<td>Harvest</td>
|
|
77 |
<td>The act (by Harvester) of visiting a Harvest Site, retrieving a
|
|
78 |
number of EML documents, and inserting or updating the documents to
|
|
79 |
Metacat.
|
|
80 |
</td>
|
|
81 |
</tr>
|
|
82 |
<tr>
|
|
83 |
<td>Harvest List</td>
|
|
84 |
<td>An XML document that lists a set of EML documents to be harvested. The
|
|
85 |
Harvest List must conform to an XML Schema,
|
|
86 |
<a href="../../lib/harvester/harvestList.xsd">harvestList.xsd</a>.
|
|
87 |
</td>
|
|
88 |
</tr>
|
|
89 |
<tr>
|
|
90 |
<td>Site Contact</td>
|
|
91 |
<td>The individual at a particular Harvest Site who registers with
|
|
92 |
Harvester, composes a Harvest List, and periodically prepares
|
|
93 |
the site's EML documents for retrieval and upload to Metacat.
|
|
94 |
</td>
|
|
95 |
</tr>
|
|
96 |
<tr>
|
|
97 |
<td>Harvest List URL</td>
|
|
98 |
<td>A URL to the Harvest List, as specified by the Site Contact.
|
|
99 |
Each Harvest Site corresponds to a Harvest List URL. Harvester
|
|
100 |
uses the URL to locate and read a site's Harvest List.
|
|
101 |
</td>
|
|
102 |
</tr>
|
|
103 |
<tr>
|
|
104 |
<td>Document URL</td>
|
|
105 |
<td>A URL to an EML document, as specified in the Harvest List.
|
|
106 |
The Harvest List may contain any number of Document URLs. Each
|
|
107 |
Document URL provides a locator to a document to be harvested.
|
|
108 |
</td>
|
|
109 |
</tr>
|
|
110 |
<tr>
|
|
111 |
<td>Harvester Registration Page</td>
|
|
112 |
<td>A web page that provides a means for a Site Contact
|
|
113 |
to register with Harvester to schedule regular harvests from the
|
|
114 |
site. Registration involves logging in and then specifying various
|
|
115 |
settings for the Harvest Site, such as the Harvest List URL, the
|
|
116 |
harvest frequency, and the email address of the Site Contact.
|
|
117 |
</td>
|
|
118 |
</tr>
|
|
119 |
</table>
|
|
120 |
<h4>Managing Harvester</h4>
|
|
121 |
Harvester is managed by the Harvester Administrator. Typically, the same
|
|
122 |
individual who manages a Metacat server would also act as the Harvester
|
|
123 |
Administrator. The responsibilities of the Harvester Administrator include:
|
|
124 |
<ul>
|
|
125 |
<li><a href="#Configuring Harvester">Configuring Harvester</a></li>
|
|
126 |
<li><a href="#Running Harvester">Running Harvester</a></li>
|
|
127 |
<li><a href="#Reviewing Harvester">Reviewing Harvester reports to
|
|
128 |
the Harvester Administrator</a></li>
|
|
129 |
</ul>
|
|
130 |
<h5><a name="Configuring Harvester">Configuring Harvester</a></h5>
|
|
131 |
<p>Harvester must be configured to interact with a working Metacat
|
|
132 |
installation. Thus, a Metacat installation that has been properly
|
|
133 |
configured and installed is a pre-requisite to running Harvester.
|
|
134 |
Additionally, Harvester has a number of settable properties that
|
|
135 |
control its behavior. All Harvester configuration information is managed
|
|
136 |
in a single file,
|
|
137 |
<a href=../../lib/harvester/harvester.properties>harvester.properties</a>,
|
|
138 |
located at:
|
|
139 |
<pre> METACAT_HOME/lib/harvester/harvester.properties</pre>
|
|
140 |
where METACAT_HOME is the top-level directory that Metacat is
|
|
141 |
installed in.
|
|
142 |
</p>
|
|
143 |
<p>The Harvester Administrator should edit
|
|
144 |
<a href=../../lib/harvester/harvester.properties>harvester.properties</a>,
|
|
145 |
setting appropriate values for the Metacat URL, database driver,
|
|
146 |
database connection, and other settings. The
|
|
147 |
following table is a summary of each property and its function.
|
|
148 |
</p>
|
|
149 |
<table border="1">
|
|
150 |
<tr>
|
|
151 |
<td><b>Property</b></td>
|
|
152 |
<td><b>Description</b></td>
|
|
153 |
<td><b>Possible or default value</b></td>
|
|
154 |
</tr>
|
|
155 |
<tr>
|
|
156 |
<td>connectToMetacat</td>
|
|
157 |
<td>This property determines whether Harvester should connect to
|
|
158 |
Metacat to upload documents. It should be set to <code>true</code>
|
|
159 |
under most circumstances. Setting this property to <code>false</code>
|
|
160 |
can be useful for testing whether Harvester is able to retrieve
|
|
161 |
documents from a site without actually connecting to Metacat to
|
|
162 |
upload the documents.</td>
|
|
163 |
<td><code>true</code> | <code>false</code><br>
|
|
164 |
Default: <code>true</code>
|
|
165 |
</tr>
|
|
166 |
<tr>
|
|
167 |
<td>dbDriver</td>
|
|
168 |
<td>The JDBC driver to be used to access the backend database. This
|
|
169 |
setting should match the value of the dbDriver property as set
|
|
170 |
in the <a href=../../build.xml>build.xml</a> file as appropriate
|
|
171 |
to the database being used (Oracle, PostgreSQL, or SQL Server).
|
|
172 |
</td>
|
|
173 |
<td>Examples:<br>
|
|
174 |
<code>oracle.jdbc.driver.OracleDriver</code><br>
|
|
175 |
<code>org.postgresql.Driver</code><br>
|
|
176 |
<code>com.microsoft.jdbc.sqlserver.SQLServerDriver</code>
|
|
177 |
</td>
|
|
178 |
</tr>
|
|
179 |
<tr>
|
|
180 |
<td>defaultDB</td>
|
|
181 |
<td>The JDBC connection string that Metacat uses to connect to the
|
|
182 |
backend database. This setting should match the value of
|
|
183 |
the <code>jdbc-connect</code> property as set in the
|
|
184 |
<a href=../../build.properties>build.properties</a>
|
|
185 |
file in the associated Metacat installation.</td>
|
|
186 |
<td>Example:<br>
|
|
187 |
<code>jdbc:oracle:thin:@server.domain.com:1521:Metacat</code></td>
|
|
188 |
</tr>
|
|
189 |
<tr>
|
|
190 |
<td>delay</td>
|
|
191 |
<td>The number of hours that Harvester will wait before beginning its
|
|
192 |
first harvest. For example, if Harvester is run at 1:00 p.m., and
|
|
193 |
the delay is set to 12, Harvester will begin its first harvest at
|
|
194 |
1:00 a.m.</td>
|
|
195 |
<td>Default: 0</td>
|
|
196 |
</tr>
|
|
197 |
<tr>
|
|
198 |
<td>harvesterAdministrator</td>
|
|
199 |
<td>The email address of the Harvester Administrator. Harvester will
|
|
200 |
send email reports to this address after every harvest.
|
|
201 |
</td>
|
|
202 |
<td>An email address</td>
|
|
203 |
</tr>
|
|
204 |
<tr>
|
|
205 |
<td>logPeriod</td>
|
|
206 |
<td>The number of days that Harvester should retain log entries of harvest
|
|
207 |
operations in the database. Harvester log entries record information
|
|
208 |
such as which documents were harvested, from which sites, and
|
|
209 |
whether any errors were encountered during the harvest. Log entries
|
|
210 |
older than <code>logPeriod</code> number of days are purged from the
|
|
211 |
database at the end of each harvest.</td>
|
|
212 |
<td>Default: 90</td>
|
|
213 |
</tr>
|
|
214 |
<tr>
|
|
215 |
<td>maxHarvests</td>
|
|
216 |
<td>The maximum number of harvests that Harvester should execute before
|
|
217 |
shutting down. When the Harvester program is executed, it will
|
|
218 |
continue running until it has executed <code>maxHarvests</code>
|
|
219 |
number of harvests and then the program will terminate.</td>
|
|
220 |
<td>Default: 30</td>
|
|
221 |
</tr>
|
|
222 |
<tr>
|
|
223 |
<td>metacatURL</td>
|
|
224 |
<td>The URL of the Metacat servlet to which Harvester should connect
|
|
225 |
for uploading documents.</td>
|
|
226 |
<td>Example:<br>
|
|
227 |
http://somehost.institution.edu:8080/knb/servlet/metacat</td>
|
|
228 |
</tr>
|
|
229 |
<tr>
|
|
230 |
<td>password</td>
|
|
231 |
<td>The password that Harvester uses to access the backend database.
|
|
232 |
This setting should match the value of the <code>password</code>
|
|
233 |
property as set in the
|
|
234 |
<a href=../../build.properties>build.properties</a>
|
|
235 |
file in the associated Metacat installation.
|
|
236 |
</td>
|
|
237 |
<td> </td>
|
|
238 |
</tr>
|
|
239 |
<tr>
|
|
240 |
<td>period</td>
|
|
241 |
<td>The number of hours between harvests. Harvester will run a new
|
|
242 |
harvest every <code>period</code> number of hours, until the
|
|
243 |
<code>maxHarvests</code> number of harvests have been run.</td>
|
|
244 |
<td>Default: 24</td>
|
|
245 |
</tr>
|
|
246 |
<tr>
|
|
247 |
<td>smtpServer</td>
|
|
248 |
<td>The SMTP server that Harvester uses for sending email messages
|
|
249 |
to the Harvester Administrator and to Site Contacts.</td>
|
|
250 |
<td>A host name, for example: <code>somehost.institution.edu</code>
|
|
251 |
<br><br>
|
|
252 |
Default: <code>localhost</code>
|
|
253 |
<br><br>
|
|
254 |
Note that the default value will only work if the Harvester
|
|
255 |
host machine has been configured as a SMTP server.
|
|
256 |
</td>
|
|
257 |
</tr>
|
|
258 |
<tr>
|
|
259 |
<td>user</td>
|
|
260 |
<td>The username that Metacat uses to access the backend database.
|
|
261 |
This setting should match the <code>user</code> value as set in the
|
|
262 |
<a href=../../build.properties>build.properties</a>
|
|
263 |
file in the associated Metacat installation.
|
|
264 |
</td>
|
|
265 |
<td> </td>
|
|
266 |
</tr>
|
|
267 |
<tr>
|
|
268 |
<td>Harvester Operation Properties (GetDocError, GetDocSuccess, etc.)</td>
|
|
269 |
<td>This group of properties is used by Harvester to report information
|
|
270 |
about the operations it performs for inclusion in log
|
|
271 |
entries and email messages. Under most circumstances the values
|
|
272 |
of these properties should not be modified.</td>
|
|
273 |
<td> </td>
|
|
274 |
</tr>
|
|
275 |
</table>
|
|
276 |
<br>
|
|
277 |
<h5><a name="Running Harvester">Running Harvester</a></h5>
|
|
278 |
After Harvester has been appropriately
|
|
279 |
<a href="#Configuring Harvester">configured</a>,
|
|
280 |
it can be run as follows:
|
|
281 |
<ol>
|
|
282 |
<li>Open a system command window or terminal window.</li>
|
|
283 |
<li>Set the METACAT_HOME environment variable to the value of the Metacat
|
|
284 |
installation directory. Some examples follow:
|
|
285 |
<ul>
|
|
286 |
<li>On Windows:
|
|
287 |
<pre>set METACAT_HOME=C:\somePath\metacat</pre></li>
|
|
288 |
<li>On Linux/Unix (bash shell):
|
|
289 |
<pre>export METACAT_HOME=/home/somePath/metacat</pre></li>
|
|
290 |
</ul>
|
|
291 |
<li>cd to the following directory:
|
|
292 |
<ul>
|
|
293 |
<li>On Windows:
|
|
294 |
<pre>cd %METACAT_HOME%\lib\harvester</pre></li>
|
|
295 |
<li>On Linux/Unix:
|
|
296 |
<pre>cd $METACAT_HOME/lib/harvester</pre></li>
|
|
297 |
</ul>
|
|
298 |
<li>Run the appropriate Harvester shell script, as determined by the
|
|
299 |
operating system:
|
|
300 |
<ul>
|
|
301 |
<li>On Windows:
|
|
302 |
<pre>runHarvester.bat</pre></li>
|
|
303 |
<li>On Linux/Unix:
|
|
304 |
<pre>sh runHarvester.sh</pre></li>
|
|
305 |
</ul>
|
|
306 |
</li>
|
|
307 |
</ol>
|
|
308 |
<p>The Harvester application will start executing. It will begin its first
|
|
309 |
harvest after <code><b>delay</b></code> number of hours (as specified in the
|
|
310 |
<a href=../../lib/harvester/harvester.properties>harvester.properties</a>
|
|
311 |
file). The application will continue running a new harvest every
|
|
312 |
<code><b>period</b></code> number of hours until a <code><b>maxHarvests</b></code>
|
|
313 |
number of harvests have been completed.
|
|
314 |
</p>
|
|
315 |
<h5><a name="Reviewing Harvester">
|
|
316 |
Reviewing Harvester Reports to the Harvester Administrator</a></h5>
|
|
317 |
<P>
|
|
318 |
After every harvest, Harvester will send an email report to the Harvester
|
|
319 |
Administrator detailing the operations that were performed during the
|
|
320 |
harvest. The report will contain information about each of the Harvest Sites
|
|
321 |
that were harvested from, such as which EML documents were
|
|
322 |
harvested and whether any errors were encountered.
|
|
323 |
</P>
|
|
324 |
<p>
|
|
325 |
The harvest report will contain a list of log entries, where each log entry
|
|
326 |
describes an operation that was performed by Harvester. Log entries that
|
|
327 |
show a status value of 1 indicate that an error occurred during the
|
|
328 |
operation, while those that show a status value of 0 indicate that the
|
|
329 |
operation was completed successfully.
|
|
330 |
</p>
|
|
331 |
<P>The Harvester Administrator should review the report, paying particularly
|
|
332 |
close attention to any errors that are reported and to the accompanying error
|
|
333 |
messages that are displayed. When errors are reported at
|
|
334 |
a particular site, the Harvester Administrator should contact the Site
|
|
335 |
Contact to determine the source of the error and its resolution. See
|
|
336 |
<a href=#Reviewing>Reviewing Harvester Reports to the Site Contact</a> for a
|
|
337 |
description of common sources of errors at a Harvest Site.
|
|
338 |
</P>
|
|
339 |
<p>Errors that are independent of a particular site may indicate a problem
|
|
340 |
with Harvester itself, Metacat, or the database connection. Refer to the
|
|
341 |
error message to determine the source of the error and its resolution.
|
|
342 |
</p>
|
|
343 |
<h4>Managing a Harvest Site</h4>
|
|
344 |
A Harvest Site is managed by a Site Contact.
|
|
345 |
The responsibilities of a Site Contact fall into the following categories:
|
|
346 |
<ul>
|
|
347 |
<li><a href=#Registering>Registering with Harvester</a></li>
|
|
348 |
<li><a href=#Composing>Composing a Harvest List</a></li>
|
|
349 |
<li><a href=#Preparing>Preparing EML Documents for harvest</a></li>
|
|
350 |
<li><a href=#Reviewing>Reviewing Harvester reports to the Site Contact</a></li>
|
|
351 |
</ul>
|
|
352 |
<h5><a name="Registering">Registering with Harvester</a></h5>
|
|
353 |
<p>
|
|
354 |
A Site Contact registers a site with Harvester by logging in to the
|
|
355 |
Harvester Registration page and entering several items of information
|
|
356 |
that Harvester needs to know about the site.
|
|
357 |
</p>
|
|
358 |
<ol>
|
|
359 |
<li>Logging in to the Harvester Registration Page
|
|
360 |
<p>
|
|
361 |
The Harvester Registration page is accessed from Metacat. For example, if
|
|
362 |
the Metacat server that you wish to register with resides at the following
|
|
363 |
URL:
|
|
364 |
<pre> http://somehost.somelocation.edu:8080/knb/index.jsp</pre>
|
|
365 |
then the Harvester Registration page would be accessed at:
|
|
366 |
<pre> http://somehost.somelocation.edu:8080/knb/style/skins/dev/harvesterRegistrationLogin.html</pre>
|
|
367 |
</p>
|
|
368 |
<p>
|
|
369 |
After bringing up this page in your browser, login to your Metacat account
|
|
370 |
by entering your username and password.
|
|
371 |
The username should include the full LDAP specification, for example:
|
|
372 |
<pre>
|
|
373 |
Username: uid=jdoe,o=lter,dc=ecoinformatics,dc=org
|
|
374 |
Password: *******
|
|
375 |
</pre>
|
|
376 |
In some cases, a Site Contact may need to login to an anonymous account
|
|
377 |
rather than his or her personal account. For example, a LTER Information
|
|
378 |
Manager may need to login to a dedicated account, named with a three-letter
|
|
379 |
acronym, that has been set up for the LTER site. For example:
|
|
380 |
<pre>
|
|
381 |
Username: uid=GCE,o=lter,dc=ecoinformatics,dc=org
|
|
382 |
Password: *******
|
|
383 |
</pre>
|
|
384 |
is the account login that would be used by the LTER Information Mangager
|
|
385 |
at the GCE (Georgia Coastal Ecosystems) site.
|
|
386 |
</p>
|
|
387 |
</li>
|
|
388 |
<li>Registering with Harvester
|
|
389 |
<p>
|
|
390 |
After logging in, you will be presented with a web form that prompts you
|
|
391 |
to enter information about your site and how often you want to schedule
|
|
392 |
harvests at your site. For example:
|
|
393 |
</p>
|
|
394 |
<pre>
|
|
395 |
Email address: myname@institution.edu
|
|
396 |
Harvest List URL: http://somehost.institution.edu/~myname/harvestList.xml
|
|
397 |
Harvest Frequency (1-99): 2
|
|
398 |
Unit: ( ) day(s) (*) week(s) ( ) month(s)
|
|
399 |
</pre>
|
|
400 |
After values have been entered for each of these fields, click the Register
|
|
401 |
button to register your site with Harvester.
|
|
402 |
</p>
|
|
403 |
<P>
|
|
404 |
In the example shown above, Harvester will attempt to harvest documents from
|
|
405 |
the site once every 2 weeks, it will access the site's Harvest List at URL
|
|
406 |
"http://somehost.institution.edu/~myname/harvestList.xml", and it will send
|
|
407 |
email reports to the Site Contact at email address "myname@institution.edu".
|
|
408 |
</P>
|
|
409 |
</li>
|
|
410 |
<li>Unregistering with Harvester
|
|
411 |
<p>
|
|
412 |
At any time after you have registered with Harvester, you may discontinue
|
|
413 |
harvests at your site by unregistering. Simply login as described above and
|
|
414 |
then click the Unregister button. After doing so, Harvester will discontinue
|
|
415 |
harvests at the site.
|
|
416 |
</p>
|
|
417 |
</li>
|
|
418 |
</ol>
|
|
419 |
<h5><a name="Composing">Composing a Harvest List</a></h5>
|
|
420 |
<p>
|
|
421 |
A Harvest List is an XML file that holds a list of EML documents to be
|
|
422 |
harvested. For each EML document in the list, the following information
|
|
423 |
must be specified:
|
|
424 |
<ul>
|
|
425 |
<li><code>docid</code>, which consists of the:
|
|
426 |
<ul>
|
|
427 |
<li><code>scope</code>, e.g. "demoDocument". The scope is an identifier
|
|
428 |
that indicates which group of documents this document belongs to.
|
|
429 |
</li>
|
|
430 |
<li><code>identifier</code>, e.g. "1". The identifier is a number that
|
|
431 |
uniquely identifies this document within the scope.
|
|
432 |
</li>
|
|
433 |
<li><code>revision</code>, e.g. "5". The revision is a number that
|
|
434 |
indicates the current revision of this document.
|
|
435 |
</li>
|
|
436 |
</ul>
|
|
437 |
</li>
|
|
438 |
<li><code>documentType</code>, e.g. "eml://ecoinformatics.org/eml-2.0.0".
|
|
439 |
The documentType identifies the document as an EML document.</li>
|
|
440 |
<li><code>documentURL</code>, e.g. "http://www.lternet.edu/~dcosta/document1.xml".
|
|
441 |
The documentURL specifies a place where Harvester can locate
|
|
442 |
and retrieve the document via HTTP.</li>
|
|
443 |
</ul>
|
|
444 |
</p>
|
|
445 |
<p>
|
|
446 |
The contents of a Harvest List XML file must conform to a particular
|
|
447 |
XML Schema, as defined in file <a href="../../lib/harvester/harvestList.xsd">
|
|
448 |
harvestList.xsd</a>. The contents of a valid Harvest List
|
|
449 |
can best be illustrated by example. The sample Harvest List
|
|
450 |
below contains two <<code>document</code>> elements that specify the
|
|
451 |
information that Harvester needs to retrieve a pair of EML documents and
|
|
452 |
upload them to Metacat:
|
|
453 |
<pre>
|
|
454 |
<?xml version="1.0" encoding="UTF-8" ?>
|
|
455 |
<hrv:harvestList xmlns:hrv="eml://ecoinformatics.org/harvestList" >
|
|
456 |
<document>
|
|
457 |
<docid>
|
|
458 |
<scope>demoDocument</scope>
|
|
459 |
<identifier>1</identifier>
|
|
460 |
<revision>5</revision>
|
|
461 |
</docid>
|
|
462 |
<documentType>eml://ecoinformatics.org/eml-2.0.0</documentType>
|
|
463 |
<documentURL>http://www.lternet.edu/~dcosta/document1.xml</documentURL>
|
|
464 |
</document>
|
|
465 |
<document>
|
|
466 |
<docid>
|
|
467 |
<scope>demoDocument</scope>
|
|
468 |
<identifier>2</identifier>
|
|
469 |
<revision>1</revision>
|
|
470 |
</docid>
|
|
471 |
<documentType>eml://ecoinformatics.org/eml-2.0.0</documentType>
|
|
472 |
<documentURL>http://www.lternet.edu/~dcosta/document2.xml</documentURL>
|
|
473 |
</document>
|
|
474 |
</hrv:harvestList>
|
|
475 |
</pre>
|
|
476 |
<p>
|
|
477 |
After editing the Harvest List, ensure that the Harvest List XML file resides
|
|
478 |
at the appropriate location on disk as specified by the URL that was entered
|
|
479 |
during the <a href=#Registering>registration</a> process.
|
|
480 |
</p>
|
|
481 |
<h5><a name="Preparing">Preparing EML Documents for harvest</a></h5>
|
|
482 |
<p>
|
|
483 |
To prepare a set of EML documents for harvest, ensure that the following is
|
|
484 |
true for each document:
|
|
485 |
<ul>
|
|
486 |
<li>The document contains valid EML</li>
|
|
487 |
<li>The document is specified in a <document> element in the
|
|
488 |
site's Harvest List, as described above</li>
|
|
489 |
<li>The file resides at the appropriate location on disk as specified
|
|
490 |
by its URL in the Harvest List</li>
|
|
491 |
</ul>
|
|
492 |
</p>
|
|
493 |
<h5><a name="Reviewing" >Reviewing Harvester Reports to the Site Contact</a></h5>
|
|
494 |
<P>
|
|
495 |
After every scheduled harvest that takes place at a particular Harvest
|
|
496 |
Site, Harvester will send an email report to the Site Contact detailing the
|
|
497 |
operations that were performed during the harvest.
|
|
498 |
The report will contain information about the operations that were
|
|
499 |
performed by Harvester at that site, such as
|
|
500 |
which EML documents were harvested and whether any errors were encountered.
|
|
501 |
</P>
|
|
502 |
<P>
|
|
503 |
The Site Contact should review the report, paying particularly
|
|
504 |
close attention to any errors that are reported. Errors are indicated
|
|
505 |
by operations that display a status value of 1, while operations that
|
|
506 |
display a status value of 0 indicate that the operation completed
|
|
507 |
successfully.
|
|
508 |
</P>
|
|
509 |
<p>
|
|
510 |
When errors are reported,
|
|
511 |
the Site Contact should try to determine whether the source of the error
|
|
512 |
is something that can be corrected at the site. Common causes of errors
|
|
513 |
might be:
|
|
514 |
<ul>
|
|
515 |
<li>A document URL specified in the Harvest List does not match
|
|
516 |
the location of the actual EML file on the disk</li>
|
|
517 |
<li>The Harvest List does not contain valid XML as specified in
|
|
518 |
the <a href=../../lib/harvester/harvestList.xsd>harvestList.xsd</a> schema</li>
|
|
519 |
<li>The URL to the Harvest List that was specified during
|
|
520 |
registration with Harvester does not match the actual location of
|
|
521 |
the Harvest List on the disk</li>
|
|
522 |
<li>An EML document that Harvester attempted to upload to Metacat does
|
|
523 |
not contain valid EML</li>
|
|
524 |
</ul>
|
|
525 |
</P>
|
|
526 |
<p>
|
|
527 |
If the Site Contact is unable to determine the cause of the error and its
|
|
528 |
resolution, he or she should contact the Harvester Administrator for assistance.
|
|
529 |
</p>
|
|
530 |
<a href="./properties.html">Back</a> |
|
|
531 |
<a href="./metacattour.html">Home</a> |
|
|
532 |
<a href="./unimplem.html">Next</a>
|
|
533 |
</BODY>
|
|
534 |
</HTML>
|
| 0 |
535 |
|
Add Harvester documentation to the Metacat Tour