Kepler

The ComadTest actor is roughly the COMAD equivalent of Ptolemy's NonStrictTest. It records the stream of data it receives during a run. In training mode it stores an XML representation (similar to a trace file) of this stream in a parameter value. In test mode it throws an exception if the received stream doesn't match the previously stored value of this parameter.

This works fine currently if the showDetails parameter is set to false, hiding all token and object IDs in the stored XML representation. However, this makes it impossible to verify that the provenance records are correct because the dep attribute is hidden (the value of the dep attribute is a list of the ids of tokens on which a new item depends). However, if the IDs are recorded then the ComadTest actor almost always reports a mismatch because, e.g., the IDs in the dep attribute are not guaranteed to be in a particular order, IDs are assigned asynchronously in different actors, etc.

We do not expect traces for multiple runs of COMAD workflows to be exactly reproducible in general due to concurrency. Each actor runs in its own thread and token IDs are pulled from a common pool. But in a simple, three-actor test workflow (CollectionComposer -> ActorToBeTested -> ComadTest), I believe reproducibility to be both possible and desirable.