Project

General

Profile

metacat / docs / user / metacat / source / ore-model-expansion.rst @ 8786

1
.. raw:: latex
2

    
3
  \newpage
4
  
5

    
6
ORE Model for Derived Data Packages
7
===================================
8

    
9
.. index:: Derived Resources
10
.. index:: PROV
11

    
12
Author
13
  Lauren Walker
14

    
15
Date
16
  - 20140513 Initial draft of ORE model expansion for derived data packages
17

    
18
Goal
19
  Accurately describe the relationships of resources that are derived from other resources.
20

    
21
Summary 
22
  This proposed expansion of the Metacat ORE model uses the `PROV ontology <http://www.w3.org/TR/prov-overview/>`_ to describe
23
  the relationships between generated or derived resources and the resources used to generate them.
24
  
25
Overview
26
---------------------------------------
27
There are two main concepts that are being introduced to Metacat - derived data resources and the activities that create them.
28

    
29
A derived resource is defined as a entity, usually a data file, that is produced by transforming one entity into another or is the production of an activity that generates a new entity from an existing one. 
30
In Metacat, derived data could be a data table that was composed from analyzing one or more existing data tables. 
31
It could also be a visualization of another resource, such as a graph, chart, or map.
32

    
33
Because derived data is the product of a transformation activity, Metacat's ORE model will need to have the capability to store information on these activities. 
34
These activities could be a script (e.g. an R script) that is used to create a visualization from a data table or to compose a new data table from one or more existing data tables. 
35

    
36
These new resources will be documented by metadata, most likely EML.
37

    
38

    
39
Ontology and Model
40
------------------
41
The `PROV ontology <http://www.w3.org/TR/prov-overview/>`_ will be used to describe the following relationships:
42
  * Derived resources and the original resources used to create them 
43
  * Derived resources and the activities used to create them
44

    
45
.. figure:: images/PROV-simple-diagram.png
46

    
47
   Figure 1. A simple diagram illustrating the PROV ontology Metacat will use
48
   
49
This model will allow the original data file to remain unchanged as all assertions made about the derived data can be 
50
described in the derived data RDF.
51

    
52

    
53
The terms from the PROV ontology that will be introduced to Metacat to describe these relationships:
54
   * `wasGeneratedBy <http://www.w3.org/ns/prov#wasGeneratedBy>`_ : Generation is the completion of production of a new entity by an activity. This entity did not exist before generation and becomes available for usage after this generation.
55
   * `generated <http://www.w3.org/ns/prov#generated>`_ : Generation is the completion of production of a new entity by an activity. This entity did not exist before generation and becomes available for usage after this generation.
56
   * `wasDerivedFrom <http://www.w3.org/ns/prov#wasDerivedFrom>`_ : A derivation is a transformation of an entity into another, an update of an entity resulting in a new one, or the construction of a new entity based on a pre-existing entity.
57
   * `used <http://www.w3.org/ns/prov#used>`_ : Usage is the beginning of utilizing an entity by an activity. Before usage, the activity had not begun to utilize this entity and could not have been affected by the entity.
58
   * `wasInformedBy <http://www.w3.org/ns/prov#wasInformedBy>`_ : Communication is the exchange of an entity by two activities, one activity using the entity generated by the other.
59
   
60
The following diagram fully represents the new model for derived resources. New terms and resources are highlighted in purple.
61

    
62
.. figure:: images/Derived-data-packages-model.png
63

    
64
   Figure 2. The model for derived data sets with new terms highlighted in purple.
65

    
66

    
67
Examples
68
---------
69
Derived data using scripts
70
__________________________
71
John Smith is a scientist who has uploaded two data tables and an EML file to a Metacat repository. His package of data is represented below.
72

    
73
.. figure:: images/derived-data-smith-package.png
74

    
75
	Figure 3. Smith's original data package using the ORE model Metacat has now
76
	
77
Jessica Couture is an analyst who wants to create a graph of John's Smith data. She creates an R script that grabs the data files smith_data.1.1 and smith_data.2.1,
78
performs some sort of analysis on them and transforms them into a single data file, couture_data.1.1. She then creates another R script to visualize this composed data
79
into a chart. She describes her process and the data in an EML file and uploads all five files to Metacat. Below is the model that describes Couture's new data package:
80

    
81
.. figure:: images/Derived-data-example.png
82

    
83
	Figure 4. Couture's derived data package with the existing Metacat ORE notation in blue and the new PROV notation in purple.
84

    
85
Derived data without scripts
86
____________________________
87
Not all derived data will be produced from a script that can be stored in Metacat. An analyst may create a composed data table through a more
88
manual process, such as transforming values in a spreadsheet program like MS Excel. Or a scientist may create a simple graph of their data with
89
MS Excel or an online graphing tool.
90

    
91
Let's say John Smith decides to create a time series chart using the data from one of his data files, smith_data.1.1. He adds this image file to
92
the original data package and edits the metadata to explain his process and the chart. His data package will essentially be using the PROV model without the activity concept.
93
Smith has the option of creating a new data package with the image and an EML file so that his metadata ID does not have to change. 
94

    
95
.. figure:: images/PROV-simple-diagram-without-activity.png
96

    
97
   Figure 4. The PROV model without an activity
98

    
99
.. figure:: images/derived-data-smith-package-with-img.png
100

    
101
   Figure 5. Smith's new data package with a data visualization. Note that the resource map and metadata have new revisions. 
102
 
103
Sample RDF
104
----------
105
A sample RDF notation for our above Couture and Smith example (Figure 4) can be :download:`downloaded <_static/resourceMap_couture.1.1.rdf>` or viewed below. Note that the 
106
primary data resource will need to be described in the RDFs for its derivations so that an RDF for a derived dataset will have all the information it needs to relate the primary
107
resource to the derived resource. This way, a SparQL query can be run within a derived dataset RDF to make the assertion that derived_metadata wasDerivedFrom primary_metadata.
108

    
109
::
110

    
111
	<rdf:RDF
112
	    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
113
	    xmlns:foaf="http://xmlns.com/foaf/0.1/"
114
	    xmlns:owl="http://www.w3.org/2002/07/owl#"
115
	    xmlns:dc="http://purl.org/dc/elements/1.1/"
116
	    xmlns:ore="http://www.openarchives.org/ore/terms/"
117
	    xmlns:dcterms="http://purl.org/dc/terms/"
118
	    xmlns:j.0="http://purl.org/spar/cito/"
119
	    xmlns:prov="http://www.w3.org/ns/prov#">
120
	    
121
	  <rdf:Description rdf:about="https://cn.dataone.org/cn/v1/resolve/couture_data.1.1">
122
	    <ore:isAggregatedBy>https://cn.dataone.org/cn/v1/resolve/resourceMap_couture.1.1#aggregation</ore:isAggregatedBy>
123
	    <dcterms:identifier rdf:datatype="http://www.w3.org/2001/XMLSchema#string">couture_data.1.1</dcterms:identifier>
124
	    <j.0:isDocumentedBy rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_metadata.1.1"/>
125
	    <prov:wasGeneratedBy rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_composeScript.1.1"/>
126
	    <prov:wasDerivedFrom rdf:resource="https://cn.dataone.org/cn/v1/resolve/smith_data.1.1"/>
127
	    <prov:wasDerivedFrom rdf:resource="https://cn.dataone.org/cn/v1/resolve/smith_data.2.1"/>
128
	  </rdf:Description>
129
	      
130
	  <rdf:Description rdf:about="https://cn.dataone.org/cn/v1/resolve/smith_data.1.1">
131
	    <ore:isAggregatedBy>https://cn.dataone.org/cn/v1/resolve/resourceMap_smith .1.1#aggregation</ore:isAggregatedBy>
132
	    <dcterms:identifier rdf:datatype="http://www.w3.org/2001/XMLSchema#string">smith_data.1.1</dcterms:identifier>
133
	    <j.0:isDocumentedBy rdf:resource="https://cn.dataone.org/cn/v1/resolve/smith_metadata.1.1"/>
134
	  </rdf:Description>
135
	    
136
	  <rdf:Description rdf:about="https://cn.dataone.org/cn/v1/resolve/resourceMap_couture.1.1#aggregation">
137
	    <rdf:type rdf:resource="http://www.openarchives.org/ore/terms/Aggregation"/>
138
	    <ore:isDescribedBy rdf:resource="https://cn.dataone.org/cn/v1/resolve/resourceMap_couture.1.1"/>
139
	    <dc:title>DataONE Aggregation</dc:title>
140
	    <ore:aggregates rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_data.1.1"/>
141
	    <ore:aggregates rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_metadata.1.1"/>
142
	    <ore:aggregates rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_img.1.1"/>
143
	    <ore:aggregates rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_script.1.1"/>
144
	    <ore:aggregates rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_composeScript.1.1"/>
145
	  </rdf:Description>
146
	    
147
	  <rdf:Description rdf:nodeID="A0">
148
	    <foaf:name rdf:datatype="http://www.w3.org/2001/XMLSchema#string">Java libclient</foaf:name>
149
	    <rdf:type rdf:resource="http://purl.org/dc/terms/Agent"/>
150
	  </rdf:Description>
151
	    
152
	  <rdf:Description rdf:about="https://cn.dataone.org/cn/v1/resolve/resourceMap_couture.1.1">
153
	    <dcterms:modified rdf:datatype="http://www.w3.org/2001/XMLSchema#dateTime">2013-09-03T09:54:06.392-07:00</dcterms:modified>
154
	    <ore:describes rdf:resource="https://cn.dataone.org/cn/v1/resolve/resourceMap_couture.1.1#aggregation"/>
155
	    <rdf:type rdf:resource="http://www.openarchives.org/ore/terms/ResourceMap"/>
156
	    <dc:creator rdf:nodeID="A0"/>
157
	    <dcterms:identifier rdf:datatype="http://www.w3.org/2001/XMLSchema#string">resourceMap_couture.1.1</dcterms:identifier>
158
	  </rdf:Description>
159
	    
160
	  <rdf:Description rdf:about="https://cn.dataone.org/cn/v1/resolve/couture_metadata.1.1">
161
	    <ore:isAggregatedBy>https://cn.dataone.org/cn/v1/resolve/resourceMap_couture.1.1#aggregation</ore:isAggregatedBy>
162
	    <dcterms:identifier rdf:datatype="http://www.w3.org/2001/XMLSchema#string">couture_metadata.1.1</dcterms:identifier>
163
	    <j.0:documents rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_data.1.1"/>
164
	    <j.0:documents rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_img.1.1"/>
165
	    <j.0:documents rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_script.1.1"/>
166
	    <j.0:documents rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_composeScript.1.1"/>
167
	  </rdf:Description>
168
	    
169
	  <rdf:Description rdf:about="https://cn.dataone.org/cn/v1/resolve/couture_img.1.1">
170
	    <ore:isAggregatedBy>https://cn.dataone.org/cn/v1/resolve/resourceMap_couture.1.1#aggregation</ore:isAggregatedBy>
171
	    <dcterms:identifier rdf:datatype="http://www.w3.org/2001/XMLSchema#string">couture_img.1.1</dcterms:identifier>
172
	    <j.0:isDocumentedBy rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_metadata.1.1"/>
173
	    <prov:wasDerivedFrom rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_data.1.1"/>
174
	    <prov:wasGeneratedBy rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_script.1.1"/>
175
	  </rdf:Description>
176
	    
177
	  <rdf:Description rdf:about="https://cn.dataone.org/cn/v1/resolve/couture_script.1.1">
178
	    <ore:isAggregatedBy>https://cn.dataone.org/cn/v1/resolve/resourceMap_couture.1.1#aggregation</ore:isAggregatedBy>
179
	    <dcterms:identifier rdf:datatype="http://www.w3.org/2001/XMLSchema#string">couture_script.1.1</dcterms:identifier>
180
	    <j.0:isDocumentedBy rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_metadata.1.1"/>
181
	    <prov:used rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_data.1.1"/>
182
	    <prov:generated rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_img.1.1"/>
183
	    <prov:wasInformedby rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_composeScript.1.1" />
184
	  </rdf:Description>
185
	    
186
	  <rdf:Description rdf:about="https://cn.dataone.org/cn/v1/resolve/couture_composeScript.1.1">
187
	    <ore:isAggregatedBy>https://cn.dataone.org/cn/v1/resolve/resourceMap_couture.1.1#aggregation</ore:isAggregatedBy>
188
	    <dcterms:identifier rdf:datatype="http://www.w3.org/2001/XMLSchema#string">couture_composeScript.1.1</dcterms:identifier>
189
	    <j.0:isDocumentedBy rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_metadata.1.1"/>
190
	    <prov:used rdf:resource="https://cn.dataone.org/cn/v1/resolve/smith_data.1.1"/>
191
	    <prov:used rdf:resource="https://cn.dataone.org/cn/v1/resolve/smith_data.2.1"/>
192
	    <prov:generated rdf:resource="https://cn.dataone.org/cn/v1/resolve/couture_data.1.1"/>
193
	  </rdf:Description>
194
	    
195
	</rdf:RDF>
196

    
197

    
198
::
199

    
200
Indexing
201
--------
202
The Metacat Index will index some of these relationships mainly to inform MetacatUI.
203

    
204
New Index Fields
205
	* ``wasGeneratedBy``
206
	* ``generated``
207
	* ``used``
208
	* ``wasDerivedFrom``
209
	* ``wasInformedBy``
210
	* ``hadDerivation``
211

    
212
All of these fields are taken directly from the PROV ontology, except for ``hadDerivation``. Because MetacatUI navigation is metadata-centric 
213
(i.e. metadata are retrieved and listed as search results and details on datasets are viewed via the metadata view [/#view]), it is helpful to 
214
use SparQL to infer relationships between metadata so that it isn't necessary to send multiple queries to the index to retrieve information about 
215
derived datasets.
216

    
217
SparQL will infer that the derived metadata wasDerivedFrom the primary source metadata and the Metacat index will index both this relationship and it's inverse, 
218
hadDerivation, for easier querying.	The SparQL query used to determine the derived metadata for a primary metadata would look like:
219

    
220
::
221

    
222
	SELECT ?primary_metadata_pid ?derived_metadata_pid
223
	WHERE {
224
		?primary_data		ore:isDocumentedBy	?primary_metadata .
225
		?primary_metadata	dcterms:identifier 	?primary_metadata_pid .
226
		?derived_data		prov:wasDerivedFrom	?primary_data .
227
		?derived_data		ore:isDocumentedBy	?derived_metadata .
228
		?derived_metadata	dcterms:identifier 	?derived_metadata_pid .
229
	}
230
	
231
::
232

    
233
More information about inverses in PROV-O and in particular, ``hadDerivation``, see `section B of the PROV documentation <http://www.w3.org/TR/prov-o/#inverse-names>`_.
234
	
235
Example
236
_______
237

    
238
Using the above Couture and Smith example (Figure 4), the following fields and values from the PROV relationships would be indexed.
239

    
240
+---------------------------+----------------+---------------------+--------------------------------+
241
| Object                    |  Field Name    | Field Type          |             Value              |
242
+===========================+================+=====================+================================+
243
| couture_metadata.1.1      | wasDerivedFrom | string, multivalued | smith_metadata.1.1             |
244
+---------------------------+----------------+---------------------+--------------------------------+
245
| smith_metadata.1.1        | hadDerivation  | string, multivalued | couture_metadata.1.1           |
246
+---------------------------+----------------+---------------------+--------------------------------+
247
| couture_img.1.1           | wasGeneratedby | string, multivalued | couture_script.1.1             |
248
+---------------------------+----------------+---------------------+--------------------------------+
249
| couture_img.1.1           | wasDerivedFrom | string, multivalued | couture_data.1.1               |
250
+---------------------------+----------------+---------------------+--------------------------------+
251
| couture_script.1.1        | used           | string, multivalued | couture_data.1.1               |
252
+---------------------------+----------------+---------------------+--------------------------------+
253
| couture_script.1.1        | wasInformedBy  | string, multivalued | couture_composeScript.1.1      |
254
+---------------------------+----------------+---------------------+--------------------------------+
255
| couture_script.1.1        | generated      | string, multivalued | couture_img.1.1                |
256
+---------------------------+----------------+---------------------+--------------------------------+
257
| couture_data.1.1          | wasGeneratedBy | string, multivalued | couture_composeScript.1.1      |
258
+---------------------------+----------------+---------------------+--------------------------------+
259
| couture_data.1.1          | wasDerivedFrom | string, multivalued | smith_data.1.1, smith_data.2.1 |
260
+---------------------------+----------------+---------------------+--------------------------------+
261
| couture_composeScript.1.1 | used           | string, multivalued | smith_data.1.1, smith_data.2.1 |
262
+---------------------------+----------------+---------------------+--------------------------------+
263
| couture_composeScript.1.1 | generated      | string, multivalued | couture_data.1.1               |
264
+---------------------------+----------------+---------------------+--------------------------------+
265

    
266
Queries
267
_______
268
These indexed fields will be used primarily by MetacatUI to determine relationships between resources. An example use case is the MetacatUI metadata view (/#view/{pid}).
269
When a user navigates to the metadata view for a primary resource, such as smith_metadata.1.1 from the above example, MetacatUI can send a query to the index to look for
270
derived resources that the user might be interested in. Some pseduocode to describe this logic:
271

    
272
::
273

    
274
	if(this_metadata.hadDerivation){
275
		hadDerivation = this_metadata.hadDerivation;
276
		sendQuery("q=isDocumentedBy:" + hadDerivation + "+-obsoletedBy:*&fl=id");
277
	}
278

    
279

    
280

    
281
This query would retrieve the IDs of objects which are documented by metadata that is derived from the metadata currently being viewed. In the above Amith and Couture example (Figure 4), the following IDs
282
would be returned from this query:
283

    
284
* ``couture_composeScript.1.1``
285
* ``couture_script.1.1``
286
* ``couture_data.1.1``
287
* ``couture_img.1.1``
288

    
289

    
290
	
291
  
292