Bug #1836


data search and access problems via ecogrid

Added by Dan Higgins over 19 years ago. Updated about 16 years ago.

data access
Target version:
Start date:
Due date:
% Done:


Estimated time:


In the recent BEAM ENM meeting, ~20 people were searching kepler/ecogrid for a
datafile (on metacat). It worked OK for some, but seemed to take a long time.
For others, nothing was returned (time out?); others got different numbers of
hits returned! (The difference appeared to be hits from DIGIR appeared for some
and didn't for others.)

Overall, performance was extremely varied and generally poor. (Maybe because
there were a lot of people hitting DIGIR and Metacat at the same time.)

Actions #1

Updated by Rod Spears over 19 years ago

One of the reasons DiGIR is slow is because we ALWAYS hit all 10 or providers
and have to wait for all the resultsets to come back before it can return any

This "should" be improved once we have metadata in the regisrty and enable the
users to be more selective about where they want to search.

If they don't want to be selective and want to get all the data from everywhere
then it will take a while. This has partically to do with setting the user's
expectations about the task they are performing. We know what it is doing behind
the scenes so we have a different set of expectations.

They have no idea the amount of data that is being searched and how it is
collected and combined and returned.

Actions #2

Updated by Rod Spears over 19 years ago

Also, as we enable more providers the slowness with only get worse.

Actions #3

Updated by Laura Downey over 19 years ago

Giving some further input on this to help the debug progess. When some of the
users couldn't get any search results returned on their machines, they moved
to a neighbor's machine (that had previously been successful in getting some
search results) and performed the search there and some users were able to get

The same problems occured when the search term was "test" or "Datos
Meteorologicos." With the latter, people tried various pieces of the search
term. Some used the entire search term, others used just "datos" or "Datos"
and still were unable to get any search results.

Still other users tried re-booting Kepler and at least one user was successful
in getting search results after doing that.

A few users tried to limit the search to just metacat and were still
unsuccessful in getting any search results returned.

Regardless of whether users are searching across a large data set or several
large data sets, their expectation was some type of results, not a time out
(or the progress indicator just stopping with no results returned). Obviously
the moving progress indicator is what lets users know that things are still
working, and it is a nice feature. However, if we anticipate even more
slowness in the future, we will need to design some additional feedback, such
as letting the user know the system is searching data set 1, then data set 2

Actions #4

Updated by Matt Jones over 18 years ago

Initial fixes introduced using the Factory method to fix some of these issues.
Need to verify that they work, and adapt to the decision to move to plain web
services in the next EcoGrid release. Then test, and close this bug. Higgins
agreed to develop a test case for this demonstrating data access failures.

Actions #5

Updated by Dan Higgins over 18 years ago

This bug is related to Bug # 2174 in SEEK Bugzilla which describes details of
problems with mutliple datasources.

A test for this is the workflows/eco/IPCC_Base_Layers.xml workflow with tries to
load 10 IPCC datasource files. This reliably 'hangs' on the last of the
datasources (which remains 'red').

Actions #6

Updated by Jing Tao about 18 years ago

After the ecogrid switch to pure axis, the issues seems get better. In the
workshop at Jan. 2006, about 30 people kept searching and getting data from
ecogrid and it worked well.

I used 3 machine to do a stress testing in ecogrid and it worked well.

But the stress testing by the cluster in NCEAS (which has 14 machines),
sometimes failed - some search didn't get result set back.

Actions #7

Updated by Dan Higgins almost 18 years ago

Although there are some performance issues, the ecogrid connection now seems much more robust. I am thus removing this general bug as FIXED. Specific other problems should be added as needed. (Dan Higgins)

Actions #8

Updated by Jing Tao about 16 years ago

In 2008 workshop in New Mexico, the ecogrid search worked fine. Dan and I think we can close this bug.

Actions #9

Updated by Redmine Admin about 11 years ago

Original Bugzilla ID was 1836


Also available in: Atom PDF