Metacat

Modified Metacat to support large data file uploads. This is accomplished
by supporting a new content type for data sent to metacat:
multipart/form-data
which allows multiple files to be sent in a standard MIME format. The
MetacatServlet.handleGetOrPost() method now checks the incoming content...

Simplified ReplicationHandler:
It is run now in the same thread started from DBSAXHandler;
thus the same connection opened in that thread is used,
i.e. no need for new connection and no complains as "Connection is closed" in ReplicationHandler;
"Connection is closed" error happens when in the current thread there are try to use connection opened in another thread,...

support for the new terms included in pathquery.dtd

http://bugzilla.ecoinformatics.org/show_bug.cgi?id=248
Included utility function about gettig the latest docid for a user:
DBUtil.getLastDocid(username).
Metacat parameters:
action='getlastdocid'
username

don't need to override the old meta_file_id tags with docid without rev
since Morpho inserts docids before sending the docs to Metacat

added constructor in DocumentImpl for use in DBQuery.findDocuments();
this new constructor is used with getDocumentInfo() to get the doc info only;
because the other constructor builds the whole xml doc which is not needed here and time consuming;
this happens on backtracking only.

fixed bug - hardcoded LDAP URL

made use of the new property for ldaps url;
it is used for secure connection to LDAP server listening on second port 636 by default with SSL sockets;
it is used from Metacat for the authetication process only;
all the rest communications with LDAP server are made on the default port 389 with plain sockets

moved delete from xml_relation before delete from xml_documents
because of the parent-child relation b/w these 2 tables

fixed error with exception handling in metacat servlet and added new dtds to the sql script

changes according to bug# 234 - metacat accession # handling, done

changed all mentions of DBAdapter to the new name AbstractDatabase

changed all mentions of DBAdapter to the new name AbstractDatabase

rename the DB* classes to:
AbstractDatabase
OracleAdapter
PostgresqlAdapter
SqlserverAdapter

use the non-null function name from db adapter

included new function that returns db specific non-null function name

excluded revisionid from the insert statement for xml_revisions.
It is identity column for SQL Server or filled by sequence for db as Oracle or Postgres

changed like with =

changed ... WHERE ... LIKE ? ...
to ... WHERE ... = ? ...

MS SQL Server stores trailing spaces in varchar fields which shouldn't be
thus added docid.trim() in the Java code

the ODBC multi-threading problem:
in DBQuery we open ResultSet in the db connection and we are trying to
open another Resultset by AccessControlList which does not work
thus open separate connection for use by AccessControlList.

changed
SELECT ... WHERE LIKE ? ...
to:
SELECT ... WHERE = ? ...

It should be changed everywhere it is found

seems that there are problem in JDBC-ODBC bridge used with SQL Server
when using sql statements like:
SELECT ...
WHERE LIKE ? ...
changed to:
SELECT ...
WHERE = ? ...
and now it works

commented out the use of oracle.jdbc.driver
this was harcoded use for batching the sql calls at once in order to speed up the insert of XML Index

Changed the order of generating and getting Unique ID for
xml_nodes.nodeid and for uniqueid part of the accession#
because of SQl Server capabilities:
first let the db to generate unique id on insert (by db trigger and sequence or by IDENTITY key),
then get that unique id for further use by the application....

- new db adapter for MS SQL Server
- changed the getUniqueID(conn, tableName) to return the last generated by the db unique ID
instead of generating unique ID first by the appl and then supplied to the insert,
because SQL Server doesn't have capabilities
unique ID to be generated on the client and then supplied to the insert.

change the call from getDateFunction() to getDateTimeFunction()

changed the getDate method to getDateTime and changed the return
string in the postgresql implementation to "now()" from now()::date

change the call from getDateString() to getDateFunction()

change the method name getDateString() to getDateFunction()
return "now()::date" in the DBPostgresql implementation of that method

The Postgres implementation of the dbadapter package, still needs testing and
the getDate... method just returns the db function to return the date; to retreive the
current time is a second function so we may want to add a second method to return the
current time.

- changed to use the DBAdapter uniform interface for all apperance of "sysdate"
and uniqueid generation from sequence
MetaCatUtil:
- reads from metacat.properties the dbadapter class implemented the interface
and creates an instance of that class (in static public variable named dbAdapter) for use....

the abstract DBAdapter class and its Oracle implementation
for the new dbadapter package

Fixed bug in metacat where return doctypes were not being properly
back traced. Changed DBQuery.findDocuments() so that it no longer
takes a returndoc array, but instead retrieves the returndoc array
fromthe QUERYSpecification. The QuerySpecification was augmented...

Added ability to process multiple "returndoc" elements in
a pathquery document. Now, any query hit will be back traced to an
associated package if that package is listed in one of the
"returndoctype" elements.

Modifications that change the package handling behavior of metacat. Package
membership is now determined by an identifier being used as either the subject
or object in a triple. One can not request arbitrary returndocs -- only
those types that have package information embedded can be "returndoctype"...

fix in the return of the last revisionid on UPDATE instead of the previous one

fixed the problem with empy password authentication.
Ldap somehow allows DirContext to be created
when empty password is provided as in our case
instead of comlaining like with wrong password string.
So included a check for empty password in order to reject the login.

as John were testing Metacat on postgres,
he discovered that Connection needs more precise check
whether it is open because of db timing outs.
So used to be:
if ( conn null ) { get new connection from db; }
Now changed to:
if ( conn null || conn.isClosed() ) { get new connection from db; }...

fix to view abstract only when abstractpath parameter was specified

put exception handler messages when acc# is not in the correct format as:
<sitecode>.<uniqueid>.[<revisionid>]

fix - added back generate(docid, action) method called from DataFileServer.java where rev is defaulted to 1

- changes to expect revisionid to come from the client on INSERT/UPDATE/DELETE;
now the client should send accession# as:
sitecode.uniqueid.revisionid
INSERT checks if sitecode.uniqueid part does not exist in xml_documents and xml_revisions and revisionid=1...

Tomcat has got configurated mime type mappings, but some are
missing. Thus added support for them in MetaCatServlet as:
File type > MIME type
------------- ----------------
.xml -> "text/xml"
.xsd -> "text/xml"
.dtd -> "text/plain"
.css -> "text/css"...

- merged "getabstract" and "getdatadoc" actions to "read" action;
- put MIME types everywhere for the returned data;
- zip when more than one doc/files were requested or when specified by qformat="zip" parameter;
- get rid of "relation" parameter, instead use "docid" parameter name only - when more that one docid - zip them;...

Changes related to running LDAP servers referred each other in one tree modeling the KNB tree.
Currently there are running LDAP server on dev that holds the KNB root and the NCEAS' s subtree.
It also refers to a subtree modeling the LTER's tree which is held by another LDAP server runnig on alpha .

fixes on getting information from LDAP services

added check to see if a replication server's IP matched it's DNS entry and the entry in the replication table

fixes around getUsers and getGroups.
"getprincipals" action can now produce output like:

<principals>
<group>
<groupname>Administrators<groupname>
<user>
<username>uid=admin,o=NCEAS, c=US<username>
</user>
</group>...

Included back getting the list of users and groups stored in auth scheme
through new action="getprincipals". No extra parameters are needed.
Any logged in users are able to get this information

included missing implementation of getUsers(), getGroups() interfaces

Modifications to allow printing of SQL with and without INDEX option.

added print info and timing info

print the nested statements sql query

included flag in DBQuery to run pathqueries with or without the index,
but for the command line only.

added timing info to main

implemented interface for download of DTD or Schema file from Metacat file system
through the params:
action="getdtdschema"
doctype

- fixed missing replication of public_access
- new attribute catalog_id added in xml_documents and xml_revisions
as a FK to xml_catalog; catalog_id is replicated also

Included check up for <!DOCTYPE ... PUBLIC/SYSTEM ... >
in order to set the validation parser at runtime
In case of:
- no DOCTYPE declaration
- <!DOCTYPE docname>
validation is turned "off"
In case of:
- <!DOCTYPE ... PUBLIC ...>
- <!DOCTYPE ... SYSTEM ...>...

reject writing DTD file if it already exists

there were more mentions of DOCTITLE attr - cleared them

cleared DOCTITLE attribute in xml_documents and xml_revisions
cleared all mentions of doctitle

added new "getaccesscontrol" action for a given docid

fixed white space problem in toXML that was preventing in line end tags from being popped from the stack.

Solving the problem with relication of access file where access files were rejected from the replication servers.
In AccessControlList check for serverCode is included,
so only on the local server the connected user is checked for having "all" permissions on the resources specified in it...

fixed some problems with database connections getting a time out error. Also updated some catch statements

fixed typo

included new servlet parameter "public" specifing public read access for the submitted document
the expected values are "yes" or "no"
public parameter is optional and if it is not specified, means "no"

fixed bug that was accidentally introduced when the change from metacatURL to URL was done.

added location info to catch statements

added precise location information (class.method) to each catch statement so that errors are more easily traced.

removed print statements

fixed typo

included implementation for public "read" access as specified from the access file

fixed introduced problem with preparedStatements

Added license terms to source code files, and cleaned up some javadoc
documentation in a few places.

Added License terms to source files, added more documentation to protocol
handler classes.

closed all preparedStatement variables

implementation for multiple <resourceIndentifier> tags under <resource>

removed dependence on MetacatURL for URL handling. Added more flexible support for identifiers in the subject and object fields. They can now be just a docid, an http url or a metacat url.

fixed bug in data file permission checking. (commited wrong file last time)

- turned on the validation in order only valid xml docs comformed to the specified dtd (if any) to be submitted in metacat
- with validation "on" white spaces are reported from ignorableWhitespace() callback, not from characters() (as with validation "off")...

added more documentation

changed getCookie to static

added support for the new abstract class) to handle data file uploading through metnewninterface (actuall

newninterface (actually an abstract class) to handle data file uploading through met

fixed error in logic. sending the filesize as an int terminated by a zero failed when the file size was greater than 256. The filesize is now sent as a string followed by a zero.

added support for the server to return the docid of the newly added data file. Also, now the data stream must be prefixed with the following information: filename <0> sessionid <0> filesize <0>. where the <0> symbol is a byte containing zero. The filesize is sent as an int and the filename and sessionid are sent as strings (in byte form).

unified getting "docid" value from metacat URLs specified in <resourceIdentifier> tags
by using MetaCatUtil.parseQuery() method in the same way as in MetaCatServlet.

fixed error handling and time out issues. Also fixed thread syncronization problem

multiple files with the same names no longer overwrite each other.

fixed minor error

added support for data file upload via a random, authenticated socket.
The action added to metacatServlet is getdataport. A client can send a getdataport request. the server returns a random port number and starts a thread to handle the request on the given random port. The thread will only accept requests from the client that has the session id equal to that of the user that made the getdataport request. the user must be authenticated and logged in to make a getdataport request. Once the port number is received by the client the connection can be made. The first two parameters in the data stream must be the filename, followed by session id. (note that it only wants the session id not the whole cookie and the "JSESSION=" must be stripped from the text.) If the session number is correct, the upload is allowed, the file is written and xml_documents is updated with the new data file information.

added support for data file upload via a random, authenticated socket.

- appling acl through access files only
- clearing around update/delete of access files - to delete the related records from both xml_access and xml_relation tables
- included check for user permissions for setting acl on the resources in the access file
- moved RelationHandler.deleteRelations(docid) down in RelationHandler obj just before the write of the new relations

fixed url error in data download function

Use the new property "accessdoctype" to check if this is access file.
If it is, run AccessControlList for wirting into xml_access and xml_relation tables.
This is the thread that writes XML Index but afterwards.

included support for submition of access files
+ storing the acl info in xml_access table and relationship records in xml_relation table like
<aclfile, "isaclfilefor", recourceIdentifier> for every resource included in the access file

added support to download a data file through metacat using a metacat url or a docid parameter.

updating a package file now updates the relations that are created from it. Also, if you delete a package file, it will delete the relations associated with it.