After reviewing CNodeService and D1NodeService prompted by Robert comparing the Hazelcast locking with the d1_synchronization locking, I've made a number of changes that will prevent locking problems:
1) Multiple methods contained try/catch blocks that would:
So the lock obtained in the first try catch would never unlock if an exception is thrown. This is fixed by using an outer try/catch/finally block for locking and unlocking to ensure the lock is released.
2) In some cases we need to lock() prior to a get() from the hzSystemMetadata, other times we don't when there's no put() subsequently involved (like getChecksum(), get(), etc.) Remove any locking in those cases.