A brief overview of how I was inspecting memory, threads, file descriptors, etc. while stress testing Metacat. This should probably be added to the Metacat Admin guide in the developers section.
Checking system limits for file descriptors
On Ubuntu, user limits are seen as:
ulimit -a
On Ubuntu, systemwide open file descriptors and max allowed can be seen using:
sysctl fs.file-nr
Monitoring open file descriptors for the tomcat process:
On Mac OS X:
while (true); do lsof -p 94375 |grep -v txt |grep -v cwd |wc -l; sleep 1; done
On Ubuntu:
while (true); do lsof -a -p `pidof java` |wc -l; sleep 1; done
To generate a lot of get() requests, set up one or more terminals (on the same machine if it has enough CPUs, or on another machine), and use curl to download a list of pids (assumes that the file 'ids' exists with one id per line):
while (true); do cnt=1; for id in `cat ids`; do echo "Downloading $id : "; cnt=$((cnt + 1)); curl http://localhost:8080/metacat/d1/mn/v1/object/$id -o mdf-${cnt}; done; done
If that causes a crash, I found that sleeping for 1 or more seconds between curl calls will slow things down enough to let Metacat keep up.
To monitor memory, threads, and object creation on the JVM, I used VisualVM that ships with Java. Locally this provides graphs and lists of threads and memory used by tomcat, and allows a capture of the memory usage of individual classes. I found it useful to start a capture of memory usage, and then filter to show just 'metacat' or just 'dataone' classes to see what happens in these respective collections of classes as the system is loaded at various levels. To run VisualVM on a remote host, one can set up jstatsd to see the threads and memory usage, and then connect to the remote host with VisualVM. To also get the object instance counts and memory usage via the sampling interface, you would also have to set up JMX on the VM.