Bug #5493
opentomcat crashes frequently
0%
Description
Tomcat frequently gets into a state where it stops serving pages and needs to be restarted. This has been happening for a while now, first on the RHEL4 server (vegbank), and now on the new Ubuntu 10.04 virtual server (vegbankvm).
Most recently the tomcat service has been manually restarted on 7/30, 8/19, 8/21, 8/27, and 9/19.
I attempted to stop the frequent crashing by increasing the allocated memory in tomcat from 2GB to 4GB, but it crashed a couple days after the change, so I restored the 2GB setting.
Updated by Michael Lee over 12 years ago
Nick is correct that this is a problem. I was hoping it was the old system that was causing this. However, it appears not to be the case.
The first places we should check are the XML export and VegBranch export functions, as these are very large operations. Loading data is also large, but we can't disable that, and it's relatively uncommon. Plus, the system is hanging when no one is loading data. I suspect that many times people ask for a huge number of files via XML or VegBranch csv (a styled XML product), and it bogs down VegBank, long after they have clicked away.
So I think I will see if I can get our development VM to crash like VegBank is crashing, by downloading large XML and VegBranch csv files. If this is the case, I suggest we disable these downloads, as they are probably only very exceedingly rarely used.
The alternate explanation is deeper cruft in the code that is slowly leaking. I do not have the expertise to find that sort of problem.
Updated by Michael Lee over 12 years ago
The VegBranch import is causing this problem, as far as I can tell. We need to limit the number of plots that we allow to be downloaded into the VegBranch format.
Updated by Michael Lee over 12 years ago
The VegBranch DOWNLOAD is causing this problem, as far as I can tell. We need to limit the number of plots that we allow to be downloaded into the VegBranch format.
My last comment was unclear. It's the DOWNLOAD from VegBank that's causing the problem.
Updated by Michael Lee almost 12 years ago
I wish I could mark this resolved, but I'm not sure that it is. I disabled the VegBranch and XML downloads on VegBank and reset tomcat once again, today. If we can get to the end of June without a nagios saying it's not responsive, we may be able to resolve this bug.
At some point, we need to add more types of data to the tried-and-true csv download.