Bug #5433
closeduse 1024m instead of 512m for reporting suite
0%
Description
Kepler has a max memory setting of 512m by default. Increasing this to 1024m when using the reporting suite enables us to increase the hsqldb.cache_scale setting to 18 (which greatly improves workflow execution time performance when provenance is on by using more memory). Without making this change kepler will lock up with heap space errors after a few executions of data intensive workflows.
Updated by Derik Barseghian over 13 years ago
Just increasing to 768m is enough. In fact the one set of tests i did (below) surprisingly shows this as having better performance than 1024.
I made the change to use 768m at r27813 (note this change is just for users of the provenance suite, not regular kepler). If the kepler jvm max mem setting is < 768, it's increased. Note this means: 1) 1st kepler session will not have this memory setting in effect, and 2) this will be annoying for a user intentionally trying to set memory <768, it will get reset each time. I don't think these are big deals though. I considered and decided against only changing to 768 if the default of 512 is in use (i.e. exactly == 512).
I ran the growingDegreeDays workflow (see bug#5429) with 200days of data on my linux box with provenance in file-mode (remove "localhost" from provenance configuration.xml). After each memory change I quit kepler, and removed KeplerData and .kepler.
512m
-------------
heap space errors on 4th run
768m
-------------
1:26, 2:01, 2:23, 3:05, 2:55, 3:14, 3:15, 4:14, 3:15, 3:38, 3:37, 5:32, 4:13, 3:57, 3:25, 5:50, 4:05, 4:17, 3:48, 7:09, 4:51, 4:31, 4:20, 7:54
total: 5815s or 96.9min
no heap space errors
1024m
-------------
1:24, 2:00, 2:33, 3:02, 2:46, 4:24, 3:04, 4:27, 3:03, 3:18, 3:13, 5:28, 4:34, 4:05, 4:08, 7:02, 4:40, 4:30, 4:43, 9:37, 7:04, 6:42, 5:10, 11:08
total: 6725s or 112.08min
no head space errors