Metacat

Updated documentation to reflect new installprocess.

Modified build.xml to properly copy the docs into the distribution, which
required some changes in the ant token filtering scheme.

Improvements to the build process that make it easier to install the DTDs
form EML. There are now three new targets in the build.xml file:

getdtd: uses cvs to get a working copy of a tagged release of eml
  instdtd: copies the DTDs from the build directory to the install directory...

Updated the SQL for loading DTD references to use the correct file
names and paths and identifiers.

updated for new eml release

Fixed the getUsers() and getPrincipals() methods so that they no longer
fail when large result sets are requested. The problem was that the
LDAP server was returning a "size limit exceeded" message when the resultset
from the query exceeded the default limit of 500 entries. Now we...

Re-enabled referrals which I had turned off for debugging purposes.

Modified AuthLdap to fix the may problems associated with group and user
queries. Now the getGroups() and getUsers() methods work as advertised,
and there is a test of each of the methods in "main" for testing purposes.
Simplified the class substantially. Fixed the getAttributes method as...

fixed error where person with 'all' permission could not update the access file.

fixed error where person with 'all' permission could not update the access file.

fixed referral catching mechanism in authLdap.ldapAuthenticate() so that it will refer through a bunch of linked servers instead of just one....I still haven't figured out why the getGroups method wont work.

fixed the error where the ldap authenticate took 15 seconds to execute

I think i have fixed the ldap referral bug. the test lter account that david made for me works, however I would like matt or someone more knowedgeable with ldap to please check my code. the changes I made are around line 200 of AuthLdap. I put a comment in the source where the code needs to be checked.

Fix for bug #309 so that Metacat will now follow LDAP referrals. Previously
the default was to ignore referrals. Now we explictly set the JNDI
Context.REFERRAL value to the value in the metacat.propert "referral".
The metacat.properties file has been modified to add the "referral" property,...

added new permission 'changepermission' and made 'all' inclusive for all permissions

fixed access control bug. the character data in the sax parser was not getting trimmed and causing problems. also fixed hard coded eml-dataset public id in the web index file

Added new beta5 version of eml-software dtd, and added to catalog.

removed the stylesheet sql script since we don't need it anymore

Added new DTD files for the new release of EML. Changed the naming
convention for all new DTDs, so the public IDs are now consistently
versioned.

Updated metacat login semantics. Now, metacat assumes the username passed
in is the 'full' distinguished name of the user. If that fails, then
instead it tries looking up the string and seeing if it can determine
what the DN is, then uses it. The preferred method of logging in via...

Fixed the function to return the max id for a given scope. Now the function
takes a parameter named 'scope' and returns the largest docid that has been
used under that scope in this metacat instance (it used to return the most
recently created docid, which is clearly different). For compatibility with...

fixed bug where the whole filename of a data file was not appended to the file input stream so metacat just returned a null document

Fixed problem with metacat handleReadAction where the OutputStream from the
response object was closed and an attempt to reopen a PrintWriter was made
on the same response, which is illegal. Now the OutputStream is converted
to a PrintWriter in order to write the XML error message back to the client....

removed errant comment that was causing an error in the xml_documents table

fixed to get groupnames for a user using any identifying name

added insert statement for eml-attribute-2.0.dtd entry in xml_catalog

for action=servercontrol&subaction=add
added function for downloading of the certificate file
from the specified URL of the source server
and uploading it onto this server

changes in replication to use https

fix around one way replication:
when server A sends document to the servers in its list of replication servers with replicate=1 in xml_replication table,
the server B received the document does NOT need the opposite check on its end.

fixed error:
writing on the replication server required the whole acc# (including the rev#)

Fixed bug in metacat where an Exception was thrown when a search returned
an empty resultset. Now we avoid the function call that threw the exeption.

Added support for multiple user interfaces by enabling style sheet
transformations to be configured from within metacat rather than
from in the database. Now, metacat uses the qformat parameter to determine
a "style set", which is mapped to an xml configuration file that...

small fix to enable resolving external entities included in the DTD;
external entities are now resolved successfully by DBEntityResolver using the specified system ids
and thus there no need of registering the relative system ids,
This is because EntityResolver uses its internal implementation to map the relative system ids to the...

Removed the requirement that the relationship for ACL associations be
"isRelatedTo". Now the relationship in the triple can be anything, like
"describes access control rules for". This doesn't affect the ACL
processing because it is based on the ACL document type, not the...

changes to store namespace prefixes separately from the local names of nodes (elements and attributes)
in xml_nodes.nodeprefix column

added nodeprefix column to xml_nodes table for storing of namespace prefixes per nodes

Revised build.xml and metacat.properties to make it easier to
install alternative stylesheets for metacat. In the process,
replaced the deprecated copydir and copyfile commands in build.xml
with appropriate "copy" commands.

added support for Namespaces.
Metacat now can store and retrieve XML documents with Namespaces.
Namespace is stored as separate record in xml_nodes table with type "NAMESPACE"
where nodename is the prefix and nodedata is the uri of the namespace.

fixed an error of NullPointerException

- when the Access file goes first before the Package file (eml-dataset-2.0)
relations are not available in xml_relation, thus updated the code
to check and run ACL also after the Package file is saved.
- cut out the rev# from subject and object in xml_relation as needed by ACL...

fixed error with out of cursors error because pstmts were not getting closed in DBQuery

look for relationship "isRelatedTo" b' that is the word that is used in the triples

updated with the new eml modules

added new dtds

fixed the compilation bug about the change of group parameter in DocumentImpl.write()

added support for multiple group membership

Changed the constructors so that the nodeRecordList is automatically
retrieved by the DocumentImpl(Connection,String) constructor. this
restores its earlier behavior, but the implementation is totally
done by DocumentImpl(Connection,String,boolean).

Added fix to DocumentImpl that showed problems when the DocumentImpl(conn)
constructor was called. Now there is a new constructor:
DocumentImpl(Connection, String, boolean)
where the boolean value is used to determine if the nodeset should be
read a t the time the cosntructor is called. if false, then...

Completely removed the socket-server feature that used to provide
file upload. It is now replaced by multipart/form-data over http.

Modified Metacat to support large data file uploads. This is accomplished
by supporting a new content type for data sent to metacat:
multipart/form-data
which allows multiple files to be sent in a standard MIME format. The
MetacatServlet.handleGetOrPost() method now checks the incoming content...

Simplified ReplicationHandler:
It is run now in the same thread started from DBSAXHandler;
thus the same connection opened in that thread is used,
i.e. no need for new connection and no complains as "Connection is closed" in ReplicationHandler;
"Connection is closed" error happens when in the current thread there are try to use connection opened in another thread,...

support for the new terms included in pathquery.dtd

http://bugzilla.ecoinformatics.org/show_bug.cgi?id=248
Included utility function about gettig the latest docid for a user:
DBUtil.getLastDocid(username).
Metacat parameters:
action='getlastdocid'
username

don't need to override the old meta_file_id tags with docid without rev
since Morpho inserts docids before sending the docs to Metacat

added the new software module

added constructor in DocumentImpl for use in DBQuery.findDocuments();
this new constructor is used with getDocumentInfo() to get the doc info only;
because the other constructor builds the whole xml doc which is not needed here and time consuming;
this happens on backtracking only.

updated script with new eml20 dtds

fixed bug - hardcoded LDAP URL

made use of the new property for ldaps url;
it is used for secure connection to LDAP server listening on second port 636 by default with SSL sockets;
it is used from Metacat for the authetication process only;
all the rest communications with LDAP server are made on the default port 389 with plain sockets

moved delete from xml_relation before delete from xml_documents
because of the parent-child relation b/w these 2 tables

fixed error with exception handling in metacat servlet and added new dtds to the sql script

changes according to bug# 234 - metacat accession # handling, done

changed all mentions of DBAdapter to the new name AbstractDatabase

changed all mentions of DBAdapter to the new name AbstractDatabase

rename the DB* classes to:
AbstractDatabase
OracleAdapter
PostgresqlAdapter
SqlserverAdapter

use the non-null function name from db adapter

included new function that returns db specific non-null function name

cleared the COLLATION specifiers on column definitions:
it is new in SQL Server 2000 and not supported in its earlier versions

excluded revisionid from the insert statement for xml_revisions.
It is identity column for SQL Server or filled by sequence for db as Oracle or Postgres

changed like with =

changed ... WHERE ... LIKE ? ...
to ... WHERE ... = ? ...

MS SQL Server stores trailing spaces in varchar fields which shouldn't be
thus added docid.trim() in the Java code

the ODBC multi-threading problem:
in DBQuery we open ResultSet in the db connection and we are trying to
open another Resultset by AccessControlList which does not work
thus open separate connection for use by AccessControlList.

changed
SELECT ... WHERE LIKE ? ...
to:
SELECT ... WHERE = ? ...

It should be changed everywhere it is found

seems that there are problem in JDBC-ODBC bridge used with SQL Server
when using sql statements like:
SELECT ...
WHERE LIKE ? ...
changed to:
SELECT ...
WHERE = ? ...
and now it works

commented out the use of oracle.jdbc.driver
this was harcoded use for batching the sql calls at once in order to speed up the insert of XML Index

added
INSERT INTO xml_replication (serverid, server, replicate) VALUES ('1', 'localhost', '0');
which was missing

several changes were needed related to the change of the unique id generation and use

Changed the order of generating and getting Unique ID for
xml_nodes.nodeid and for uniqueid part of the accession#
because of SQl Server capabilities:
first let the db to generate unique id on insert (by db trigger and sequence or by IDENTITY key),
then get that unique id for further use by the application....

- new db adapter for MS SQL Server
- changed the getUniqueID(conn, tableName) to return the last generated by the db unique ID
instead of generating unique ID first by the appl and then supplied to the insert,
because SQL Server doesn't have capabilities
unique ID to be generated on the client and then supplied to the insert.

- made xmltables-sqlserver.sql from SQL Server running here on rincon machine
- included db index on xml_nodes.nodename which was missing

change the call from getDateFunction() to getDateTimeFunction()

changed the getDate method to getDateTime and changed the return
string in the postgresql implementation to "now()" from now()::date

change the call from getDateString() to getDateFunction()

change the method name getDateString() to getDateFunction()
return "now()::date" in the DBPostgresql implementation of that method

The Postgres implementation of the dbadapter package, still needs testing and
the getDate... method just returns the db function to return the date; to retreive the
current time is a second function so we may want to add a second method to return the
current time.

changed the db sequence name used for the accession# uniqueid
to xml_documents_id_seq.
All db sequences now are named of form <table_name>_id_seq

- changed to use the DBAdapter uniform interface for all apperance of "sysdate"
and uniqueid generation from sequence
MetaCatUtil:
- reads from metacat.properties the dbadapter class implemented the interface
and creates an instance of that class (in static public variable named dbAdapter) for use....

the abstract DBAdapter class and its Oracle implementation
for the new dbadapter package

Fixed bug in metacat where return doctypes were not being properly
back traced. Changed DBQuery.findDocuments() so that it no longer
takes a returndoc array, but instead retrieves the returndoc array
fromthe QUERYSpecification. The QuerySpecification was augmented...

Added ability to process multiple "returndoc" elements in
a pathquery document. Now, any query hit will be back traced to an
associated package if that package is listed in one of the
"returndoctype" elements.

Modifications that change the package handling behavior of metacat. Package
membership is now determined by an identifier being used as either the subject
or object in a triple. One can not request arbitrary returndocs -- only
those types that have package information embedded can be "returndoctype"...

fix in the return of the last revisionid on UPDATE instead of the previous one

this is sql script does the same as the sql script named
xmltables.sql except that this script is to be use to
create the database tables on a Postgresql backend rather
than an Oracle Backend

the foreign-key constraints still need to be fixed

fixed the problem with empy password authentication.
Ldap somehow allows DirContext to be created
when empty password is provided as in our case
instead of comlaining like with wrong password string.
So included a check for empty password in order to reject the login.

as John were testing Metacat on postgres,
he discovered that Connection needs more precise check
whether it is open because of db timing outs.
So used to be:
if ( conn null ) { get new connection from db; }
Now changed to:
if ( conn null || conn.isClosed() ) { get new connection from db; }...

fix to view abstract only when abstractpath parameter was specified

put exception handler messages when acc# is not in the correct format as:
<sitecode>.<uniqueid>.[<revisionid>]

fix - added back generate(docid, action) method called from DataFileServer.java where rev is defaulted to 1

- changes to expect revisionid to come from the client on INSERT/UPDATE/DELETE;
now the client should send accession# as:
sitecode.uniqueid.revisionid
INSERT checks if sitecode.uniqueid part does not exist in xml_documents and xml_revisions and revisionid=1...

Tomcat has got configurated mime type mappings, but some are
missing. Thus added support for them in MetaCatServlet as:
File type > MIME type
------------- ----------------
.xml -> "text/xml"
.xsd -> "text/xml"
.dtd -> "text/plain"
.css -> "text/css"...