1
|
#
|
2
|
# '$RCSfile$'
|
3
|
# '$Author: sgarg $'
|
4
|
# '$Revision: 2505 $'
|
5
|
# '$Date: 2005-05-24 17:24:37 -0700 (Tue, 24 May 2005) $'
|
6
|
#
|
7
|
|
8
|
Metacat: XML Metadata and Data Management System
|
9
|
------------------------------------------------
|
10
|
|
11
|
Version: 1.5.0 Release
|
12
|
|
13
|
Send feedback and bugs to: metacat-dev@ecoinformatics.org
|
14
|
http://bugzilla.ecoinformatics.org
|
15
|
|
16
|
Contributors: Matt Jones (jones@nceas.ucsb.edu)
|
17
|
Chad Berkley (berkley@nceas.ucsb.edu)
|
18
|
Jing Tao (tao@nceas.ucsb.edu)
|
19
|
Jivka Bojilova (bojilova@nceas.ucsb.edu)
|
20
|
Dan Higgins (higgins@nceas.ucsb.edu)
|
21
|
Saurabh Garg (sgarg@nceas.ucsb.edu)
|
22
|
Duane Costa (dcosta@lternet.edu)
|
23
|
Veronique Connolly (connolly@nceas.ucsb.edu)
|
24
|
|
25
|
Patch contributors:
|
26
|
Andrea Chadden (chadden@nceas.ucsb.edu)
|
27
|
Chris Jones (cjones@msi.ucsb.edu)
|
28
|
Johnoel Ancheta (johnoel@hawaii.edu)
|
29
|
Owen Jones (owen.jones@imperial.ac.uk)
|
30
|
|
31
|
Metacat is a flexible database storage system for XML formatted
|
32
|
documents and associated binary files. Metacat models the structure of the
|
33
|
XML document, therefore allowing storage of XML documents with arbitrary
|
34
|
schemas. Metacat supports several databases on the back-end, including Oracle,
|
35
|
PostgreSQL and SQL Server (limited). The server supports XML document
|
36
|
storage, retrieval, query, validation, and transformation (using the XSLT
|
37
|
stylesheet language). In addition, there is a mechanism for replicating the
|
38
|
database to other metacat instances for backup and load balancing purposes.
|
39
|
Metacat currently uses an LDAP database for authenticating users, but was
|
40
|
written to accomodate other authentication services as well.
|
41
|
|
42
|
Installation
|
43
|
------------
|
44
|
See the file "docs/user/metacatinstall.html" for detailed instructions.
|
45
|
|
46
|
Release Notes for 1.5.0:
|
47
|
------------------------
|
48
|
|
49
|
This release comes with performace improvement, user interface enhancements,
|
50
|
and bug fixes. Below is the list of bug fixes and performance improvements.
|
51
|
|
52
|
Bug Fixes:
|
53
|
* Modify property values of harvester registration servlets to match the
|
54
|
servlet-mapping URL values in web.xml. The old values used the servlet
|
55
|
class names. This worked in Tomcat 4 but seems to break in Tomcat 5 on
|
56
|
Windows. The new values use the servlet-mapping URL values. This should
|
57
|
work in both Tomcat 4 and Tomcat 5. (Bug 162)
|
58
|
* Re-implement logic to prune old log entries from the HARVEST_LOG and
|
59
|
HARVEST_DETAIL_LOG tables. The old logic caused integrity constraint
|
60
|
violations in the database because it tried to delete parent records from
|
61
|
HARVEST_LOG prior to deleting child records from
|
62
|
HARVEST_DETAIL_LOG. (Bug 162)
|
63
|
* In data registry, minor bugs fixed like the error page shows the link back
|
64
|
to the form when error page was generated because of a document read
|
65
|
request from search results page, the successful delete button leading
|
66
|
to a blank screen and the old ACL overwritten by the registry.
|
67
|
(Bugs 1307, 1310, 1322, 1344, 1551)
|
68
|
* Changed the code to support insertion of following characters: ', `,
|
69
|
charcters in the Numeric Character Reference format (&#xyz;) and
|
70
|
characters like µ. Characters like µ will be converted to and stored in
|
71
|
Numeric Character Reference format(&#xyz;). They will also be returned
|
72
|
in Numeric Character Reference and it is upto the client to convert them
|
73
|
and display them accordingly in the application. Partial fix provided by
|
74
|
Johnoel Ancheta. (Bug 1538,1711)
|
75
|
* Spatial search failed on Oracle because of invalid entries made by
|
76
|
some documents in the nodedata column of xml_nodes for paths involving
|
77
|
'eastBoundingCoordinate', 'westBoundingCoordinate',
|
78
|
'northBoundingCoordinate', 'southBoundingCoordinate'. A new column
|
79
|
nodedatanumerical was added to xml_nodes table such that all the numerical
|
80
|
nodedata is stored in this column and this column is now used for spatial
|
81
|
search. (Bug 1703, 1718)
|
82
|
* In the default skin, clicking on the keywords on the home page did not
|
83
|
result is a search being done. Fix provided by Owen Jones. (Bug 1768)
|
84
|
* Metacat generated a success message even when a document which didnt
|
85
|
exsist was requested for deletion. This has been fixed. (Bug 1850)
|
86
|
* "packagetype" in xml-relation table was entered as eml200 namespace when
|
87
|
an eml201 document was inserted. This is fixed now so that "packagetype"
|
88
|
in xml-relation table points to eml201 namespace. (Bug 1979)
|
89
|
* System_id in xml_catalog pointed to http://knb.ecoinformatics.org/knb/
|
90
|
earlier. Fixed such that it points to the locally installed metacat url.
|
91
|
(Bug 1986)
|
92
|
* Changes to the Query subsystem fix bugs that prevented attributes from
|
93
|
being expressed solely in the xpath query statement and the returnfield
|
94
|
values. For instance, a query URL may now include search strings like
|
95
|
'@packageId=sbclter%25' and return strings like 'returnfield=@packageId'.
|
96
|
Previously, the attribute had to be appended to an element:
|
97
|
'/eml/@packageId=sbclter%25'. These fixes change DBQuery.java,
|
98
|
QuerySpecification.java, and QueryTerm.java (Bug 2052)
|
99
|
* Search query failed in case of Oracle if number of documents in the result
|
100
|
were more than 1000. This has been fixed. (Bug 2000)
|
101
|
* Removed any occurence of enum which is now a keyword in Java 1.5
|
102
|
|
103
|
|
104
|
Enhancements:
|
105
|
* Major improvements done in user interface for the data registeries and
|
106
|
various other skins. Improvements done based on suggestions from Mark
|
107
|
Stromberg, Laura Downey and others. Improvements also done in resultset.xsl,
|
108
|
the default skin and ldap templates. (Bug 1948)
|
109
|
* Coded added such that administrator can delete documents irrespective of
|
110
|
who the document belongs to.
|
111
|
* Performance impovement done in searching of datasets. A part of the search
|
112
|
time was taken up because of generation of resultset after the search had be
|
113
|
done. So this was the sequence of events in earlier metacat
|
114
|
1) Search for the given text
|
115
|
2) A list of docids generated
|
116
|
3) For each of the docids
|
117
|
3a) Use xml_index and xml_nodes table to generate the string descibing
|
118
|
the document including the returnfields requested in the search
|
119
|
4) Add all the strings from step 3a to send back the resultant
|
120
|
document. Here a decent amount of time was being taken by step 3a.
|
121
|
The algorithm is now modified by addition of two tables xml_queryresult and
|
122
|
xml_returnfields and a user defined parameter xml_returnfield_count. The
|
123
|
new algorithm works as follows:
|
124
|
1) Search for the given text
|
125
|
2) A list of docids is generated
|
126
|
3) For the given set of return fields generate a unique string and check
|
127
|
if that string exists in xml_returnfields
|
128
|
3a) If string does not exist in xml_returnfield, then enter a new
|
129
|
record usage_count as 1 and returnfield_string as the unique string
|
130
|
generated above.
|
131
|
3b) Else if the string exists, increment usage_count
|
132
|
4) Given the docids from step 2 and the id of the returnfield record from
|
133
|
step 3, query for any docids that already exist in xml_queryresult.
|
134
|
For the docids that do exist, get the queryresult_string.
|
135
|
5) For rest of the docids from step2 which were not found in step 4, do
|
136
|
the following for each of the documents:
|
137
|
5a) Use xml_index and xml_nodes table to generate the string descibing
|
138
|
the document including the returnfields requested in the search
|
139
|
5b) If usage_count of the record from step is more than
|
140
|
xml_returnfield_count set in metacat.properties, then store the
|
141
|
string in xml_queryresult as a record which has the returnfield_id
|
142
|
representing the set of returnfields, docid representing the
|
143
|
document and the string as queryresult_string.
|
144
|
6) Add all the strings from step 4 and step 5a to send back the resultant
|
145
|
document
|
146
|
So the results from step 3a in previous algorithm are effectively cached
|
147
|
and hence same computation is not done again and again for each search.
|
148
|
When a document is deleted, all the entries for that document in
|
149
|
xml_queryresult table are also deleted. When a document is updated, all
|
150
|
the entries for that document in xml_queryresult table are deleted. This
|
151
|
works fine because those entries will be generated and cached again the
|
152
|
next time the document is part of a search is requested.
|
153
|
* Performance impovement done for % search.
|
154
|
* Following new functions added to metacat client API: a method to set access
|
155
|
on an xml document and a method to get newest version for a given document.
|
156
|
* Implement a new HarvesterServlet for running Harvester as a servlet. This
|
157
|
eliminates the need to run Harvester in a terminal window. By default, the
|
158
|
HarvesterServlet is commented out in lib/web.xml.tomcat(3,4,5). The user
|
159
|
documentation will be modified to instruct Harvester administrators to
|
160
|
uncomment the HarvesterServlet entry.
|
161
|
* Minor enhancement to support multiple email addresses for harvester
|
162
|
administrator and site contact. Each address is separated by a comma or
|
163
|
semicolon.
|
164
|
* Increase number of rows in Harvest List Editor from 300 to 1200.
|
165
|
* Changed default maxHarvests value to 0. Added logic to ignore maxHarvests
|
166
|
value when it is set to 0 or a negative number. This allows Harvester to
|
167
|
run indefinitely without shutting down after reaching a maximum number of
|
168
|
harvests. The previous default value of 30 would cause Harvester to
|
169
|
terminate after 30 harvests.
|
170
|
|
171
|
Details of all changes can be found in the bug database here:
|
172
|
http://bugzilla.ecoinformatics.org/buglist.cgi?product=Metacat&target_milestone=1.5
|
173
|
|
174
|
|
175
|
Release Notes for 1.4.0:
|
176
|
------------------------
|
177
|
|
178
|
This is a major release, and is the first to introduce many new features to
|
179
|
Metacat. Some of the major new features in this version of Metacat include:
|
180
|
|
181
|
* Added a new 'Harvester' that allows documents to be periodically pulled
|
182
|
from distributed sources into metacat, useful for interfacing with other
|
183
|
system types. See the Metacat Tour for details.
|
184
|
* Added new 'skins' capability to allow the GUI for the metacat web interface
|
185
|
to be more easily customized to fit into site needs. The skin is based on
|
186
|
a flexible mix of XSLT, CSS, and Javascript.
|
187
|
* Added a web-based metadata entry form for submitting limited EML documents
|
188
|
as part of a data registry. The GUI for the registry is customizable and
|
189
|
is integrated with the 'skins' system for the main metacat interface. The
|
190
|
registry subsystem is written in Perl, and can simultaneously be used to
|
191
|
present multiple 'registry' interfaces.
|
192
|
* A new 'client API' has been developed and implemented in Java and in Perl
|
193
|
so that metacat functions can be accessed from any program in those
|
194
|
languages. The client API supports the major metacat servlet actions,
|
195
|
including login(), logout(), query(), insert(), update(), delete(), and
|
196
|
upload(). See the Metacat Tour for details.
|
197
|
* Added a new 'getlog' action that produces a usage history for all major
|
198
|
events. So now an XML report can be generated for document reads, inserts,
|
199
|
deletes, etc. See the new section in the Metacat Tour.
|
200
|
* Added a new 'buildindex' action that can rebuild the index entries for any
|
201
|
document in the database.
|
202
|
* The configuration file for metacat (metacat.properties) has been moved out
|
203
|
of the jar file and into the WEB-INF directory, allowing far easier changes
|
204
|
to the configuration parameters.
|
205
|
* Results in default web interface are sorted by title in the XSLT
|
206
|
(configurable)
|
207
|
* Many bugs were fixed.
|
208
|
|
209
|
Details of all changes can be found in the bug database here:
|
210
|
http://bugzilla.ecoinformatics.org/buglist.cgi?product=Metacat&target_milestone=1.4
|
211
|
|
212
|
Release Notes for 1.3.1:
|
213
|
------------------------
|
214
|
|
215
|
This is a simple interim bug fix. No major functionality changes. Bugs fixed
|
216
|
include:
|
217
|
1) Metacat 1.3.0 doesn't work in Java 1.3 because a subtle API differnce in
|
218
|
Java 1.3 and Java 1.4. Currently, Metacat 1.3.1 will work in both Java 1.3
|
219
|
and Java 1.4.
|
220
|
2) Distribution package size was reduced.
|
221
|
|
222
|
|
223
|
Release Notes for 1.3.0
|
224
|
------------------------
|
225
|
|
226
|
In 1.3.0 release, the structure of the xml_access table was changed and a new
|
227
|
table, xml_accesssubtree was added. If you try to update a previously
|
228
|
installed Metacat, you should run a script file to updated the table structure
|
229
|
before installation.
|
230
|
For Oracle user: At the SQLPLUS prompt type the following -
|
231
|
@src/reviseformetacat13.sql
|
232
|
For Postgresql user: At install directory prompt type the following -
|
233
|
psql exp < src/reviseformetacat13_postgres.sql
|
234
|
(where 'exp' is the database name).
|
235
|
|
236
|
After installation, user should run "ant schemasql" command to register EML2
|
237
|
schema in xml_catalog table.
|
238
|
|
239
|
If you are a new Metacat user, this step is unneeded.
|
240
|
|
241
|
Note: 1) We recommend to use Tomcat 4 and JAVA 1.4 to run Metacat. Otherwise,
|
242
|
it will cause potential replication issues.
|
243
|
2) Delete the xercesImpl.jar and xmlParserAPIs.jar files
|
244
|
which are in $CATALINA_HOME/common/endorsed. They are old version and
|
245
|
don't support XML schema validation.
|
246
|
|
247
|
New Features in 1.3.0
|
248
|
1) Partialy support EML2 document. User can store, query, read and write
|
249
|
EML2 documents. However, Metacat only support access control in resource
|
250
|
level. The subtree level access control will be ignored.
|
251
|
2) Support other xml document base on namespace/schema.
|
252
|
3) Support query for attribute value
|
253
|
4) Assign MIME type to data file base on metadata when user try to read it.
|
254
|
5) Owner can assign access rules to a document which does not have access
|
255
|
document to apply it.
|
256
|
6) Support exporting single file, not only whole package
|
257
|
7) Resupport Microsoft SQL Server.
|
258
|
|
259
|
Fixes in 1.3.0:
|
260
|
1) Couldn't finish delta T replication for large set of documents.
|
261
|
2) Couldn't create access control during delta T replication.
|
262
|
3) Eorr will be written to a seperated log file if some documents
|
263
|
were failed in replication.
|
264
|
4) Decrease the time to create access rules during insert or update
|
265
|
a package.
|
266
|
|
267
|
Documentation
|
268
|
-------------
|
269
|
See the docs directory for detailed documentation and installation
|
270
|
instructions.
|
271
|
|
272
|
Details of the Metacat architecture can be found on the website for
|
273
|
the Knowledge Network for Biocomplexity (KNB):
|
274
|
|
275
|
http://knb.ecoinformatics.org/software/metacat/
|
276
|
|
277
|
Contributions to this work are welcome. Please see the above web site
|
278
|
for details on how to contribute.
|
279
|
|
280
|
Major Known Bugs or Feature Requests (see http://bugzilla.ecoinformatics.org)
|
281
|
-----------------------------------------------------------------------------
|
282
|
If you discover a bug please report it, either by email (above) or by using
|
283
|
our bug tracking system (http://bugzilla.ecoinformatics.org). There is a
|
284
|
list of currently unimplemented features in Bugzilla that we are working on
|
285
|
for the next release.
|
286
|
|
287
|
Legalese
|
288
|
--------
|
289
|
This software is copyrighted by The Regents of the University of California
|
290
|
and the National Center for Ecological Analysis and Synthesis
|
291
|
and licensed under the GNU GPL; see the 'LICENSE' file for
|
292
|
details.
|
293
|
|
294
|
This material is based upon work supported by the
|
295
|
National Science Foundation under Grant No. DEB99-80154, DBI99-04777, and
|
296
|
0225676. Any opinions, findings and conclusions or recomendations expressed
|
297
|
in this material are those of the author(s) and do not necessarily reflect
|
298
|
the views of the National Science Foundation (NSF).
|
299
|
|
300
|
This software is partially supported by a grant from the Andrew W.
|
301
|
Mellon Foundation.
|
302
|
|
303
|
This product includes software developed by the Apache Software
|
304
|
Foundation (http://www.apache.org/). See the LICENSE file in lib/apache
|
305
|
for details.
|
306
|
|
307
|
The source code, object code, and documentation in the com.oreilly.servlet
|
308
|
package is copyright and owned by Jason Hunter. See the cos-license.html file
|
309
|
for details of the license. Licensor retains title to and ownership of the
|
310
|
Software and all enhancements, modifications, and updates to the Software.
|
311
|
|