Project

General

Profile

Bug #5796

D1Object in Morpho still uses byte array as data source

Added by Jing Tao almost 8 years ago. Updated almost 8 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
morpho - general
Target version:
Start date:
01/18/2013
Due date:
% Done:

0%

Estimated time:
Bugzilla-Id:
5796

Description

When morpho reads and saves the data package to the DataONE network, it handle the D1Objects. The entities, metadata and ore document will be transformed to the D1Object. However, the D1Object is still using the byt arrary as data source. That means we may put a big data file into the memory. This may cause some memory issue. The orignal design is that the D1Object will have a class attribute - a DataStore interface. The DataStore interface get and set methods. Those methods are handling InputStream and OutputStream, rather than byte array.

History

#1 Updated by ben leinfelder almost 8 years ago

We discussed the following changes to D1Object.
-add: getDataInputStream() and setDataInputStream()
-modify: getData() and setData() to operate without actually storing the byte[]
-use the stream-based methods in Morpho.

D1Object will likely defer to the D1Client (with cache enabled) to retrieve the data input stream initially.

#2 Updated by Jing Tao almost 8 years ago

I am thinking about the two methods we will add (in comment 1):

getDataInputStream() and setDataInputStream()

I think we should call getDataOutputStream()

So we should have a way to persistent the data in D1Object.

I believe the DataStore interface concept is still very useful. You may find it at;
https://repository.dataone.org/documents/Projects/cicore/architecture/api-documentation/source/design/morpho/images/morpho-class-diagram.png

DataStore has two methods:

OutputStream get(Identifier id);
void set(Identifier id, InputStream input);

#3 Updated by Jing Tao almost 8 years ago

create an issue on the dataone redmine. So the dataone persons can know what we will do.
https://redmine.dataone.org/issues/3523

#4 Updated by Jing Tao almost 8 years ago

Now we use DataSource for D1Object to replace the byte[]

#5 Updated by Redmine Admin over 7 years ago

Original Bugzilla ID was 5796

Also available in: Atom PDF