Metacat

Modified Metacat to support large data file uploads. This is accomplished
by supporting a new content type for data sent to metacat:
multipart/form-data
which allows multiple files to be sent in a standard MIME format. The
MetacatServlet.handleGetOrPost() method now checks the incoming content...

Simplified ReplicationHandler:
It is run now in the same thread started from DBSAXHandler;
thus the same connection opened in that thread is used,
i.e. no need for new connection and no complains as "Connection is closed" in ReplicationHandler;
"Connection is closed" error happens when in the current thread there are try to use connection opened in another thread,...

tests made on Oracle and SQL Server.

support for the new terms included in pathquery.dtd

included new terms for searchmode:
equals (former matches-exactly)
isnot-equal
less-than
greater-than
less-than-equals
greater-than-equals

http://bugzilla.ecoinformatics.org/show_bug.cgi?id=248
Included utility function about gettig the latest docid for a user:
DBUtil.getLastDocid(username).
Metacat parameters:
action='getlastdocid'
username

don't need to override the old meta_file_id tags with docid without rev
since Morpho inserts docids before sending the docs to Metacat

added the new software module

added constructor in DocumentImpl for use in DBQuery.findDocuments();
this new constructor is used with getDocumentInfo() to get the doc info only;
because the other constructor builds the whole xml doc which is not needed here and time consuming;
this happens on backtracking only.

updated script with new eml20 dtds

fixed bug - hardcoded LDAP URL

made use of the new property for ldaps url;
it is used for secure connection to LDAP server listening on second port 636 by default with SSL sockets;
it is used from Metacat for the authetication process only;
all the rest communications with LDAP server are made on the default port 389 with plain sockets

new property for ldaps:// url for LDAP connection over SSL;
it is set on second port - by default 636;
for plain LDAP communications the default port 389 is used.

moved delete from xml_relation before delete from xml_documents
because of the parent-child relation b/w these 2 tables

cleared the text under the figure; should go in the Word text

EPS files saved as AdobeIllustrator v7.0

EPS file for the IC paper

fixed error with exception handling in metacat servlet and added new dtds to the sql script

changes according to bug# 234 - metacat accession # handling, done

changed all mentions of DBAdapter to the new name AbstractDatabase

changed all mentions of DBAdapter to the new name AbstractDatabase

changed dbAdapter property to edu...OracleAdapter

rename the DB* classes to:
AbstractDatabase
OracleAdapter
PostgresqlAdapter
SqlserverAdapter

use the non-null function name from db adapter

included new function that returns db specific non-null function name

cleared the COLLATION specifiers on column definitions:
it is new in SQL Server 2000 and not supported in its earlier versions

excluded revisionid from the insert statement for xml_revisions.
It is identity column for SQL Server or filled by sequence for db as Oracle or Postgres

changed like with =

changed ... WHERE ... LIKE ? ...
to ... WHERE ... = ? ...

MS SQL Server stores trailing spaces in varchar fields which shouldn't be
thus added docid.trim() in the Java code

the ODBC multi-threading problem:
in DBQuery we open ResultSet in the db connection and we are trying to
open another Resultset by AccessControlList which does not work
thus open separate connection for use by AccessControlList.

changed
SELECT ... WHERE LIKE ? ...
to:
SELECT ... WHERE = ? ...

It should be changed everywhere it is found

seems that there are problem in JDBC-ODBC bridge used with SQL Server
when using sql statements like:
SELECT ...
WHERE LIKE ? ...
changed to:
SELECT ...
WHERE = ? ...
and now it works

commented out the use of oracle.jdbc.driver
this was harcoded use for batching the sql calls at once in order to speed up the insert of XML Index

added
INSERT INTO xml_replication (serverid, server, replicate) VALUES ('1', 'localhost', '0');
which was missing

several changes were needed related to the change of the unique id generation and use

Changed the order of generating and getting Unique ID for
xml_nodes.nodeid and for uniqueid part of the accession#
because of SQl Server capabilities:
first let the db to generate unique id on insert (by db trigger and sequence or by IDENTITY key),
then get that unique id for further use by the application....

- new db adapter for MS SQL Server
- changed the getUniqueID(conn, tableName) to return the last generated by the db unique ID
instead of generating unique ID first by the appl and then supplied to the insert,
because SQL Server doesn't have capabilities
unique ID to be generated on the client and then supplied to the insert.

Updated pathquery.dtd to reflect new addition of filterdoctype element.

- made xmltables-sqlserver.sql from SQL Server running here on rincon machine
- included db index on xml_nodes.nodename which was missing

change the call from getDateFunction() to getDateTimeFunction()

changed the getDate method to getDateTime and changed the return
string in the postgresql implementation to "now()" from now()::date

change the call from getDateString() to getDateFunction()

change the method name getDateString() to getDateFunction()
return "now()::date" in the DBPostgresql implementation of that method

The Postgres implementation of the dbadapter package, still needs testing and
the getDate... method just returns the db function to return the date; to retreive the
current time is a second function so we may want to add a second method to return the
current time.

changed the db sequence name used for the accession# uniqueid
to xml_documents_id_seq.
All db sequences now are named of form <table_name>_id_seq

added new property for the dbadapter class as:
dbAdapter=edu.ucsb.nceas.dbadapter.DBOracle
the installation procedure should change it to the appropriate class

- changed to use the DBAdapter uniform interface for all apperance of "sysdate"
and uniqueid generation from sequence
MetaCatUtil:
- reads from metacat.properties the dbadapter class implemented the interface
and creates an instance of that class (in static public variable named dbAdapter) for use....

the abstract DBAdapter class and its Oracle implementation
for the new dbadapter package

Fixed bug in metacat where return doctypes were not being properly
back traced. Changed DBQuery.findDocuments() so that it no longer
takes a returndoc array, but instead retrieves the returndoc array
fromthe QUERYSpecification. The QuerySpecification was augmented...

Added ability to process multiple "returndoc" elements in
a pathquery document. Now, any query hit will be back traced to an
associated package if that package is listed in one of the
"returndoctype" elements.

Modifications that change the package handling behavior of metacat. Package
membership is now determined by an identifier being used as either the subject
or object in a triple. One can not request arbitrary returndocs -- only
those types that have package information embedded can be "returndoctype"...

fix in the return of the last revisionid on UPDATE instead of the previous one

this is sql script does the same as the sql script named
xmltables.sql except that this script is to be use to
create the database tables on a Postgresql backend rather
than an Oracle Backend

the foreign-key constraints still need to be fixed

fixed the problem with empy password authentication.
Ldap somehow allows DirContext to be created
when empty password is provided as in our case
instead of comlaining like with wrong password string.
So included a check for empty password in order to reject the login.

as John were testing Metacat on postgres,
he discovered that Connection needs more precise check
whether it is open because of db timing outs.
So used to be:
if ( conn null ) { get new connection from db; }
Now changed to:
if ( conn null || conn.isClosed() ) { get new connection from db; }...

fix to view abstract only when abstractpath parameter was specified

specified the correct param list about the "read" action changes

put exception handler messages when acc# is not in the correct format as:
<sitecode>.<uniqueid>.[<revisionid>]

fix - added back generate(docid, action) method called from DataFileServer.java where rev is defaulted to 1

- changes to expect revisionid to come from the client on INSERT/UPDATE/DELETE;
now the client should send accession# as:
sitecode.uniqueid.revisionid
INSERT checks if sitecode.uniqueid part does not exist in xml_documents and xml_revisions and revisionid=1...

Tomcat has got configurated mime type mappings, but some are
missing. Thus added support for them in MetaCatServlet as:
File type > MIME type
------------- ----------------
.xml -> "text/xml"
.xsd -> "text/xml"
.dtd -> "text/plain"
.css -> "text/css"...

chaged the hidden input box from "relation" to "docid"

- merged "getabstract" and "getdatadoc" actions to "read" action;
- put MIME types everywhere for the returned data;
- zip when more than one doc/files were requested or when specified by qformat="zip" parameter;
- get rid of "relation" parameter, instead use "docid" parameter name only - when more that one docid - zip them;...

Changes related to running LDAP servers referred each other in one tree modeling the KNB tree.
Currently there are running LDAP server on dev that holds the KNB root and the NCEAS' s subtree.
It also refers to a subtree modeling the LTER's tree which is held by another LDAP server runnig on alpha .

uncompleted script file to create DB on SQL server

fixes on getting information from LDAP services

added check to see if a replication server's IP matched it's DNS entry and the entry in the replication table

fixes around getUsers and getGroups.
"getprincipals" action can now produce output like:

<principals>
<group>
<groupname>Administrators<groupname>
<user>
<username>uid=admin,o=NCEAS, c=US<username>
</user>
</group>...

Included back getting the list of users and groups stored in auth scheme
through new action="getprincipals". No extra parameters are needed.
Any logged in users are able to get this information

removed download.xsl entry

included missing implementation of getUsers(), getGroups() interfaces

no message

no message

Added results of running the query analysis.

Analysis code cleanup.

modified analysis to add query_method (index or nested) as a
factor in the GLM.

Added data and statistical analysis code used for evaluation of the
query performance metrics for Metacat.

results from metacat testing

test results for 11thousands, 87thousands and 162 thousands nodes

play dtd pathquery files

script and data files for performance testing

removed old script and xml files

Modifications to allow printing of SQL with and without INDEX option.

added print info and timing info

print the nested statements sql query

script to execute all queries

test queries for performance analysis

included flag in DBQuery to run pathqueries with or without the index,
but for the command line only.

added timing info to main

added removal of relations

added new stylesheet for download

stylesheet for download files

interface for viewing only download documents

interface for viewing only download documents

implemented interface for download of DTD or Schema file from Metacat file system
through the params:
action="getdtdschema"
doctype

- new attribute catalog_id added in xml_documents and xml_revisions
as a FK to xml_catalog

- fixed missing replication of public_access
- new attribute catalog_id added in xml_documents and xml_revisions
as a FK to xml_catalog; catalog_id is replicated also

Added contact mailing address for bug reports and feedback.

Included check up for <!DOCTYPE ... PUBLIC/SYSTEM ... >
in order to set the validation parser at runtime
In case of:
- no DOCTYPE declaration
- <!DOCTYPE docname>
validation is turned "off"
In case of:
- <!DOCTYPE ... PUBLIC ...>
- <!DOCTYPE ... SYSTEM ...>...