data search and access problems via ecogrid
In the recent BEAM ENM meeting, ~20 people were searching kepler/ecogrid for a
datafile (on metacat). It worked OK for some, but seemed to take a long time.
For others, nothing was returned (time out?); others got different numbers of
hits returned! (The difference appeared to be hits from DIGIR appeared for some
and didn't for others.)
Overall, performance was extremely varied and generally poor. (Maybe because
there were a lot of people hitting DIGIR and Metacat at the same time.)
#1 Updated by Rod Spears over 14 years ago
One of the reasons DiGIR is slow is because we ALWAYS hit all 10 or providers
and have to wait for all the resultsets to come back before it can return any
This "should" be improved once we have metadata in the regisrty and enable the
users to be more selective about where they want to search.
If they don't want to be selective and want to get all the data from everywhere
then it will take a while. This has partically to do with setting the user's
expectations about the task they are performing. We know what it is doing behind
the scenes so we have a different set of expectations.
They have no idea the amount of data that is being searched and how it is
collected and combined and returned.
#3 Updated by Laura Downey over 14 years ago
Giving some further input on this to help the debug progess. When some of the
users couldn't get any search results returned on their machines, they moved
to a neighbor's machine (that had previously been successful in getting some
search results) and performed the search there and some users were able to get
The same problems occured when the search term was "test" or "Datos
Meteorologicos." With the latter, people tried various pieces of the search
term. Some used the entire search term, others used just "datos" or "Datos"
and still were unable to get any search results.
Still other users tried re-booting Kepler and at least one user was successful
in getting search results after doing that.
A few users tried to limit the search to just metacat and were still
unsuccessful in getting any search results returned.
Regardless of whether users are searching across a large data set or several
large data sets, their expectation was some type of results, not a time out
(or the progress indicator just stopping with no results returned). Obviously
the moving progress indicator is what lets users know that things are still
working, and it is a nice feature. However, if we anticipate even more
slowness in the future, we will need to design some additional feedback, such
as letting the user know the system is searching data set 1, then data set 2
#4 Updated by Matt Jones over 13 years ago
Initial fixes introduced using the Factory method to fix some of these issues.
Need to verify that they work, and adapt to the decision to move to plain web
services in the next EcoGrid release. Then test, and close this bug. Higgins
agreed to develop a test case for this demonstrating data access failures.
#5 Updated by Dan Higgins over 13 years ago
This bug is related to Bug # 2174 in SEEK Bugzilla which describes details of
problems with mutliple datasources.
A test for this is the workflows/eco/IPCC_Base_Layers.xml workflow with tries to
load 10 IPCC datasource files. This reliably 'hangs' on the last of the
datasources (which remains 'red').
#6 Updated by Jing Tao over 13 years ago
After the ecogrid switch to pure axis, the issues seems get better. In the
workshop at Jan. 2006, about 30 people kept searching and getting data from
ecogrid and it worked well.
I used 3 machine to do a stress testing in ecogrid and it worked well.
But the stress testing by the cluster in NCEAS (which has 14 machines),
sometimes failed - some search didn't get result set back.