Metacat

.. raw:: latex

  \newpage

Identifier Management

=====================

.. index:: Identifiers

Author

  Matthew B. Jones

Date

  - 20100301 [MBJ] Initial draft of Identifier documentation

Goal

  Extend Metacat to support identifiers with arbitrary syntax

Summary 

  Metacat currently supports identifier strings called 'docids' that have

  the syntax 'scope.object.revision', such as 'foo.34.1' (we will refer to

  these as 'LocalIDs'). We now want Metacat to support identifiers that are 

  arbitrary strings, but still enforce uniqueness and proper revision

  handling (refer to these as GUIDs).  Metacat must be able to accept 

  these strings as identifiers for all CRUD operations, and reference them 

  in search results.

Identifier Resolution

---------------------

Because Metacat uses LocalIDs throughout the code for references to objects,

and that LocalID has a constrained structure that includes semantics about

revisions in the identifier, it is difficult to wholesale replace it with

less-constrained string identifiers without re-writing much of Metacat.

Thus, our alternate strategy is to wrap the Metacat APIs with a

identifier resolution layer that keeps track of the unconstrained GUIDs and

maps them to constrained local identifiers which are used internally within

Metacat. The basic identifer table model is shown in Figure 1, while the

basic strategy for retrieving an object is shown in Figure 2, creating an 

object is shown in Figure 3, updating an object in Figure 4, and deleting

an object is shown in Figure 5.

Identifier Table Structure

~~~~~~~~~~~~~~~~~~~~~~~~~~

.. figure:: images/identifiers.png

   Figure 1. Table structure for identifiers.

..

  This block defines the table structure diagram referenced above.

  @startuml images/identifiers.png

  identifiers "*" -- "1" xml_documents

  identifiers : String identifier

  identifiers : String docid

  identifiers : Integer rev

  xml_documents : String docid

  xml_documents : String rev

  note right of identifiers

    "identifiers.(docid,rev) is a foreign key into xml_documents"

  end note

  @enduml

.. raw:: latex

  \newpage

.. raw:: pdf

  PageBreak

Handling document read operations

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

An overview of the process needed to read an object using a GUID.

.. figure:: images/guid_read.png

   Figure 2. Basic handling for string identifiers (GUIDs) as mapped to

   docids (LocalIDs) to retrieve an object.

..

  @startuml images/guid_read.png

  !include plantuml.conf

  actor User

  participant "Client" as app_client << Application >>

  participant "CRUD API" as c_crud << MetacatRestServlet >>

  participant "Identifier Manager" as ident_man << IdentifierManager >>

  participant "Handler" as handler << MetacatHandler >>

  User -> app_client

  app_client -> c_crud: get(token, GUID)

  c_crud -> ident_man: getLocalID(GUID)

  c_crud <-- ident_man: localID

  c_crud -> handler: handleReadAction(localID)

  c_crud <-- handler: object

  c_crud --> app_client: object

  @enduml

Handling document create operations

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

An overview of the process needed to create an object using a GUID.

.. figure:: images/guid_insert.png

   Figure 3. Basic handling for string identifiers (GUIDs) as mapped to

   docids (LocalIDs) to create an object.

..

  @startuml images/guid_insert.png

  !include plantuml.conf

  actor User

  participant "Client" as app_client << Application >>

  participant "CRUD API" as c_crud << MetacatRestServlet >>

  participant "Identifier Manager" as ident_man << IdentifierManager >>

  participant "Handler" as handler << MetacatHandler >>

  User -> app_client

  app_client -> c_crud: create(token, GUID, object, sysmeta)

  c_crud -> ident_man: identifierExists(GUID)

  c_crud <-- ident_man: T or F 

  alt identifierExists == "F"

      c_crud -> ident_man: mapToLocalId(GUID)

      c_crud <-- ident_man: localID

      c_crud -> handler: handleInsertAction(localID)

      c_crud <-- handler: success

      note right of c_crud

        "Also need to address how to handle the sysmeta information wrt insertion methods"

      end note

      app_client <-- c_crud: success

  else identifierExists == "T"

      app_client <-- c_crud: IdentifierNotUnique

  end

  @enduml

Handling document update operations

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

An overview of the process needed to update an object using a GUID.

.. figure:: images/guid_update.png

   Figure 4. Basic handling for string identifiers (GUIDs) as mapped to

   docids (LocalIDs) to update an object.

..

  @startuml images/guid_update.png

  !include plantuml.conf

  actor User

  participant "Client" as app_client << Application >>

  participant "CRUD API" as c_crud << MetacatRestServlet >>

  participant "Identifier Manager" as ident_man << IdentifierManager >>

  participant "Handler" as handler << MetacatHandler >>

  User -> app_client

  app_client -> c_crud: update(token, GUID, object, obsoletedGUID, sysmeta)

  c_crud -> ident_man: identifierExists(obsoletedGUID)

  c_crud <-- ident_man: T or F 

  alt identifierExists == "T"

      c_crud -> ident_man: identifierExists(GUID)

      c_crud <-- ident_man: T or F 

      alt identifierExists == "F"

          c_crud -> ident_man: mapToLocalId(GUID, obsoletedGUID)

          c_crud <-- ident_man: localID

          c_crud -> handler: handleUpdateAction(localID)

          c_crud <-- handler: success

          note right of c_crud

            "Also need to address how to handle the sysmeta information wrt update methods"

          end note

          app_client <-- c_crud: success

      else identifierExists == "T"

          app_client <-- c_crud: IdentifierNotUnique

      end

  else identifierExists == "F"

      app_client <-- c_crud: NotFound

  end

  @enduml

Handling document delete operations

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

An overview of the process needed to delete an object using a GUID.

.. figure:: images/guid_delete.png

   Figure 5. Basic handling for string identifiers (GUIDs) as mapped to

   docids (LocalIDs) to delete an object.

..

  @startuml images/guid_delete.png

  !include plantuml.conf

  actor User

  participant "Client" as app_client << Application >>

  participant "CRUD API" as c_crud << MetacatRestServlet >>

  participant "Identifier Manager" as ident_man << IdentifierManager >>

  participant "Handler" as handler << MetacatHandler >>

  User -> app_client

  app_client -> c_crud: delete(token, GUID)

  c_crud -> ident_man: identifierExists(GUID)

  c_crud <-- ident_man: T or F 

  alt identifierExists == "T"

      c_crud -> ident_man: mapToLocalId(GUID)

      c_crud <-- ident_man: localID

      c_crud -> handler: handleDeleteAction(localID)

      c_crud <-- handler: success

      app_client <-- c_crud: success

  else identifierExists == "F"

      app_client <-- c_crud: NotFound

  end

  @enduml

..

  This block defines the interaction diagram referenced above.

  startuml images/01_interaction.png

    !include plantuml.conf

    actor User

    participant "Client" as app_client << Application >>

    User -> app_client

    participant "CRUD API" as c_crud << Coordinating Node >>

    activate c_crud

    app_client -> c_crud: resolve(GUID, auth_token)

    participant "Authorization API" as c_authorize << Coordinating Node >>

    c_crud -> c_authorize: isAuth(auth_token, GUID)

    participant "Verify API" as c_ver << Coordinating Node >>

    c_authorize -> c_ver: isValidToken (token)

    c_authorize <-- c_ver: T or F

    c_crud <-- c_authorize: T or F

    app_client <-- c_crud: handle_list

    deactivate c_crud

    participant "CRUD API" as m_crud << Member Node >>

    activate m_crud

    app_client -> m_crud: get(auth_token, handle)

    participant "Server Authentication API" as m_authenticate << Member Node >>

    m_crud -> m_authenticate: isAuth(auth_token, GUID)

    m_crud <-- m_authenticate: T or F

    m_crud -> m_crud: log(get, UserID, GUID)

    app_client <-- m_crud: object or unauth or doesNotExist

    deactivate m_crud

  enduml

Metacat Administrator's Guide

=============================

.. sidebar:: Version: 2.0.0 Release

    .. image:: themes/readable/static/metacat-logo.png

       :height: 130pt

    Send feedback and bugs to: 

        metacat-dev@ecoinformatics.org

        http://bugzilla.ecoinformatics.org

    License: GPL

Metacat is a repository for data and metadata (data about data), which helps scientists find, understand and effectively use the data sets they manage or that have been created by others. Thousands of data sets are currently documented in a standardized way and stored in Metacat systems, providing the scientific community with a broad range of science data that--because the data are well and consistently described--can be easily searched, compared, merged, or used in other ways.  

- Metacat `Administrators Guide`_

- Download Metacat

    - Binary Distribution (A war file installation)

        - GZIP File: metacat-bin-2.0.0.tar.gz_

        - ZIP File: metacat-bin-2.0.0.zip_

    - Source Distribution (Full source, requiring build)

        - GZIP File: metacat-src-2.0.0.tar.gz_

        - ZIP File: metacat-src-2.0.0.zip_

    - `Older versions`_

.. _Administrators Guide: http://knb.ecoinformatics.org/software/metacat/MetacatAdministratorGuide.pdf

.. _metacat-bin-2.0.0.tar.gz: http://knb.ecoinformatics.org/software/dist/metacat-bin-2.0.0.tar.gz

.. _metacat-bin-2.0.0.zip: http://knb.ecoinformatics.org/software/dist/metacat-bin-2.0.0.zip

.. _metacat-src-2.0.0.tar.gz: http://knb.ecoinformatics.org/software/dist/metacat-src-2.0.0.tar.gz

.. _metacat-src-2.0.0.zip: http://knb.ecoinformatics.org/software/dist/metacat-src-2.0.0.zip

.. _Older versions: http://knb.ecoinformatics.org/software/dist/

Contents

========

.. toctree::

   :numbered:

   :maxdepth: 2

   01-intro.txt

   02-contributors.txt

..   03-install.txt

..   04-configuration.txt

..   05-submitting.txt

..   06-geoserver.txt

..   07-replication.txt

..   08-harvester.txt

..   09-event-logging.txt

..   10-sitemaps.txt

..   11-authinterface.txt

..   12-metacat-properties.txt

..   13-development.txt

Indices and tables

==================

* :ref:`genindex`

* :ref:`search`

Appendix: Metacat Properties

============================

Under construction!

Heading 1

------------

Heading 2

------------

Accessing and Submitting Metadata and Data

==========================================

Under construction!

Heading 1

------------

Heading 2

------------

Replication

===========

Under construction!

Heading 1

------------

Heading 2

------------

Creating a Java Class that Implements AuthInterface

===================================================

Under construction!

Heading 1

------------

Heading 2

------------

Harvester and Harvest List Editor

=================================

Under construction!

Heading 1

------------

Heading 2

------------

Metacat's Use of Geoserver

==========================

Under construction!

Heading 1

------------

Heading 2

------------

Appendix: Development Issues

============================

Metacat is an open source application, and we welcome contributions to the

source from members of the community who are interested in helping out.  This

section will contain a series of technical details about Metacat that will help

contributors understand and extend Metacat.  Much of the detail needed here is

contained in the Administrator's guide, which we will gradually migrate to this

format.

Contents:

.. toctree::

   :maxdepth: 2

   identifiers.txt

Introduction

============

Metacat is a repository for data and metadata (data about data), which helps

scientists find, understand and effectively use the data sets they manage or

that have been created by others. Thousands of data sets are currently

documented in a standardized way and stored in Metacat systems, providing the

scientific community with a broad range of science data that--because the

data are well and consistently described--can be easily searched, compared,

merged, or used in other ways.

Not only is the Metacat repository a reliable place to store metadata and data

(the database is replicated over a secure connection so that every record is

stored on multiple machines and no data is ever lost to technical failures), it

provides a user-friendly interface for information entry and retrieval.

Scientists can search the repository via the Web using a customizable search

form. Searches return results based on user-specified criteria, such as desired

geographic coverage, taxonomic coverage, and/or keywords that appear in places

such as the data set's title or owner's name. Users need only click a linked

search result to open the corresponding data-set documentation in a browser

window and discover whom to contact to obtain the data themselves (or how to

immediately download the data via the Web).

Metacat's user-friendly Registry application allows data providers to enter

data-set documentation into Metacat using a Web form. When the form is

submitted, Metacat compiles the provided documentation into the required format

and saves it. Information providers need never work directly with the XML

format in which the data are stored or with the database records themselves. In

addition, the Metacat application can easily be extended to provide a

customized data-entry interface that suits the particular requirements of each

project. Metacat users can also choose to enter metadata using the Morpho

application, which provides data-entry wizards that guide information providers

through the process of documenting each data set.

The metadata stored in Metacat includes all of the information you and others

need to understand what the described data are and how to use them: a

descriptive data set title; an abstract; the temporal, spatial, and taxonomic

coverage of the data; the data collection methods; distribution information;

and contact information. Each information provider decides who has access to

this information (the public, or just specified users), and whether or not to

upload the data set itself with the data documentation. Information providers

can also edit the metadata or delete it from the repository, again using

Metacat's straightforward Web interface.

Metacat is a Java servlet application that runs on Window or Linux platforms in

conjunction with a database, such as PostgreSQL (or Oracle 8i), and a Web

server. The Metacat application stores data in an XML format using Ecological

Metadata Language (EML) or another ecological metadata standard. For more

information about Metacat or for examples of projects currently using Metacat,

please see http://knb.ecoinformatics.org.

What's in this Guide

--------------------

The Administrator guide includes information for installing, configuring,

managing and extending Metacat for both Linux and Windows systems. Chapter Two

contains instructions for downloading and installing Metacat and the

applications required to run the software on Linux and Microsoft platforms.

Chapter Three covers how to configure Metacat, both for new and upgraded

installations. Chapter Four details the ways in which you can customize the

Metacat interface so users can access and submit information easily: using

Metacat's generic web-interface (the Registry), creating your own HTML forms,

and creating your own desktop client (like Morpho). Chapter Five discusses how

to work with Metacat's Geoserver. Chapter Six describes how to set up the

Metacat's replication service, which permits Metacat servers to share data with

each other, effectively backing up metadata and data files. Chapter Seven looks

at the Metacat Harvester, a program that automates the retrieval of EML

documents from one or more sites and their subsequent upload (insert or update)

to Metacat. Chapter Eight discusses logging, Chapter Nine contains instructions

for creating a site map, which makes individual metadata entries available via

Web searches. Metacat's Java API is included as an appendix at the end of the

guide.

Metacat Features

----------------

Metacat is a repository for metadata (data about data), which help scientists

find, understand and effectively use the data sets they manage or that have

been created by others. Specifically,

* Metacat is a Java servlet application, which can run on both Windows and Linux systems

* Metadata submitted to Metacat is broken into modules, which are stored to optimize rapid information retrieval

* Metacat's Web interface facilitates the input and retrieval of data (Figure 1.1)

* Metacat's optional mapping functionality enables you to query and visualize the geographic coverage of stored documents

* Metacat's replication feature ensures that all Metacat data and metadata is stored safely on multiple Metacat servers

* The Metacat interface can be easily extended and customized via Web forms, skins, and/or user-developed Java clients

* The Metacat harvester automates the process of retrieving and storing EML documents from one or more sites

* Metacat can be customized to use Life Sciences Identifiers (LSIDs), uniquely identifying every data record

* Metacat has a built-in logging system for tracking events such as document insertions, updates, deletes, and reads

* The appearance of Metacat's Web interface can be customized via skins. 

Contributors

============

Metacat has been designed and built by a large number of contributors to this

open source project.  Main developers and additional patch contributors are

listed here.

Contributors

------------

  - Matt Jones (jones@nceas.ucsb.edu)

  - Chad Berkley (berkley@nceas.ucsb.edu)

  - Jing Tao (tao@nceas.ucsb.edu)

  - Jivka Bojilova (bojilova@nceas.ucsb.edu)

  - Dan Higgins (higgins@nceas.ucsb.edu)

  - Saurabh Garg (sgarg@nceas.ucsb.edu)

  - Duane Costa (dcosta@lternet.edu)

  - Veronique Connolly (connolly@nceas.ucsb.edu)

  - Chris Jones (cjones@msi.ucsb.edu)

  - John Harris (harris@nceas.ucsb.edu)

  - Callie Bowdish (bowdish@ecoinformatics.org)

  - Will Tyburczy (tyburczy@ecoinformatics.org)

  - Matthew Perry (perry@nceas.ucsb.edu)

  - Chad Burt (underbluewaters@gmail.com)

  - Ben Leinfelder (leinfelder@nceas.ucsb.edu)

  - Chris Barteau (barteau@nceas.ucsb.edu)

  - Shaun Walbridge (walbridge@nceas.ucsb.edu)

  - Michael Daigle (daigle@nceas.ucsb.edu)

Patch contributors

------------------

  - Andrea Chadden (chadden@nceas.ucsb.edu)

  - Johnoel Ancheta (johnoel@hawaii.edu)

  - Owen Jones (owen.jones@imperial.ac.uk)

Enabling Web Searches: Sitemaps

===============================

Under construction!

Heading 1

------------

Heading 2

------------

Configuring Metacat

===================

Under construction!

Heading 1

------------

Heading 2

------------

Downloading and Installing Metacat

==================================

Under construction!

System Requirements

-------------------

Installing on Linux

-------------------

Quick Start Overview

~~~~~~~~~~~~~~~~~~~~

Downloading Metacat

~~~~~~~~~~~~~~~~~~~

Download the Metacat Installer

..............................

Download Metacat Source Code

............................

Check Out Metacat Source Code from SVN (for Developers)

.......................................................

Installing and Configuring Required Software

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Java 6

......

Apache Jakarta-Tomcat

.....................

Apache Web Server (Highly Recommended)

......................................

PostgreSQL Database (or Oracle 8i)

..................................

Installing and Configuring Oracle 8i

....................................

Apache Jakarta-Ant (if building from Source)

............................................

Installing Metacat

~~~~~~~~~~~~~~~~~~

Optional Installation Options (LSID Server)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Troubleshooting

~~~~~~~~~~~~~~~

Installing on Windows

---------------------

Event Logging

=============

Under construction!

Heading 1

------------

Heading 2

------------

source_suffix = '.txt'

copyright = u'2011, Regents of the University of California'