Metacat

Closed file handles in DocumentImpl and ReplicationService in commit r8297.

Also added a close() to the finally block in DocumentImpl.readFromFileSystem() in r8298. This is probably not causing the issue but won't hurt either.

Switched to using IOUtils.closeQuietly() which handles nulls and already closed file handles in r8299.

After looking into this, its not just file handles that are left open, but many InputStream and OutputStream instances of various sorts. The code needs a full audit to find all of these issues and fix them.

Checked all instances of FileInputReader and ensured that close() was being called properly. See commit r8304.

A brief overview of how I was inspecting memory, threads, file descriptors, etc. while stress testing Metacat. This should probably be added to the Metacat Admin guide in the developers section.

Checking system limits for file descriptors
On Ubuntu, user limits are seen as:

      ulimit -a

      sysctl fs.file-nr

Monitoring open file descriptors for the tomcat process:
On Mac OS X:

       while (true); do lsof -p 94375 |grep -v txt |grep -v cwd |wc -l; sleep 1; done

       while (true); do lsof -a -p `pidof java` |wc -l; sleep 1; done

To generate a lot of get() requests, set up one or more terminals (on the same machine if it has enough CPUs, or on another machine), and use curl to download a list of pids (assumes that the file 'ids' exists with one id per line):

    while (true); do cnt=1; for id in `cat ids`; do echo "Downloading $id : "; cnt=$((cnt + 1)); curl http://localhost:8080/metacat/d1/mn/v1/object/$id -o mdf-${cnt}; done; done

If that causes a crash, I found that sleeping for 1 or more seconds between curl calls will slow things down enough to let Metacat keep up.

To monitor memory, threads, and object creation on the JVM, I used VisualVM that ships with Java. Locally this provides graphs and lists of threads and memory used by tomcat, and allows a capture of the memory usage of individual classes. I found it useful to start a capture of memory usage, and then filter to show just 'metacat' or just 'dataone' classes to see what happens in these respective collections of classes as the system is loaded at various levels. To run VisualVM on a remote host, one can set up jstatsd to see the threads and memory usage, and then connect to the remote host with VisualVM. To also get the object instance counts and memory usage via the sampling interface, you would also have to set up JMX on the VM.