Kepler

 Hi All,

I have been working on trying to understand some of the details of the R system (http://www.r-project.org/) and how it might be integrated into Kepler. For those who are unfamiliar with R, "R is a system for statistical computation and graphics. It consists of a language plus a run-time environment with graphics, a debugger, access to certain system functions, and the ability to run programs stored in script files." (from the "R FAQ"). R is a powerful system for statistical and other calculations. It is comparable to Matlab or SAS but has the advantage of being free, easily extended, and available for PCs, Macs (OS X), and Unix systems. There are also numerous extensions from a variety of sources. It thus appears to be fairly widely accepted and used by numerous researchers.

A first-cut on building an R actor would seem to be to use a local version of R (since it can be freely installed on almost any computer) and run it as a sub-process to Kepler. An obvious method for doing this is to use one of the CommandLine/Exec actors.

I say 'one of ...' because there are at least 2 existing actors for running arbitrary subprocesses from within Kepler/Ptolemy.  The "CommandLine" actor can be found in the the Kepler graph editor tree under "actors/kepler/spa/CommandLine". The author listed in the source is Ilkay Altintas, and this actor runs under the 3.0.2 version of Ptolemy/Kepler. A second similar actor, called "Exec" is included with the Ptolemy 4.0Beta release under "MoreLibraries/Esoteric/Exec". The Exec actor was written for Ptolemy 4 by Chris Brooks and (I think) uses some new features that are not available in version 3.0.2. [Specifically, there is an "Expert Mode" for setting additional parameters.]

Both the CommandLine and Exec actors use the Java 'exec' method to launch a subprocess. They differ in the details, however. CommandLine actually  launches a command processor ('cmd.exe/command.exe' on Windows and 'sh' on Mac/Linux) so that  the command entered by a user is essentially identical to that entered in a terminal window to launch a process. This can include I/O redirection like "< myfile.in". In the Exec actor, the command follows the underlying  Java method more closely and has ports for input and output streams. The command string cannot include redirection. Both actors wait for the subprocess to finish before their 'fire' action completes.

Now consider just how we might integrate R into Kepler. R can be run in an interactive mode (start up; type a command; see response; type another command) or in a batch mode (start R with a script file which has a series of command and write the results to an output file). Creating an R workflow in the batch mode is fairly easy. A screen shot of a workflow which uses the CommandLine actor to run R to create a jpeg plot and then display it shown below.

The script file used in the example is:

x <- seq(-10, 10, length = 50)

y <- x

rotsinc <- function(x, y) {

    sinc <- function(x) {

        y <- sin(x)/x

        y[is.na(y)] <- 1

        y

    }

    10 * sinc(sqrt(x^2 + y^2))

}

sinc.exp <- expression(z == Sinc(sqrt(x^2 + y^2)))

z <- outer(x, y, rotsinc)   

jpeg(filename = "RTest.jpg", width = 480, height = 480, pointsize = 12,

     quality = 75, bg = "white")

par(bg = "white")

persp(x, y, z, theta = 30, phi = 30, expand = 0.5, col = "lightblue")

It can be seen in this batch approach that one can get the results from an R calculation from the output stream or from a file created by R that is then read by other Kepler actors. A problem comes up, however, if one considers how to dynamically input instructions/data to R. In batch mode, this could require the dynamic creation of script files, although it would be nicer if ports for inputing data/instructions existed for an R actor. One thus has the question of how to import information from other parts of a workflow to an R actor.

And what about using R in an interactive mode? Both the CommandLine actor and the Exec actor start a subprocess and then wait for it to finish. This means that the R code is loaded, executed, and then removed from memory.  For an interactive environment (or for the case where the R calculation is repeatedly executed). it would be desirable to only load R once!  There doesn't seem to any reason why  the R process has to be stopped between firings. One could keep the process in memory (a static variable?) and simply read the input stream, execute it, write the output to the output stream, and then wait for the next input as part of a fire event.  [Or perhaps there needs to be some class level R actor and a set of instances that do certain calculations by communicating with the class actor???]

In any case, it is possible to simulate an interactive R session using save/load workspace options when starting and ending an R session. But it would be useful if the CommandLine actor had an 'inport' port to receive commands. Also, it might be useful if the Exec actor really had input and output streams instead of the String tokens currently used (to handle long inputs).

That ends these semi-random thoughts for now.

Any comments or suggestions?

Dan

-- 

*******************************************************************

Dan Higgins                                  higgins@nceas.ucsb.edu

http://www.nceas.ucsb.edu/    Ph: 805-892-2531

National Center for Ecological Analysis and Synthesis (NCEAS) 

735 State Street - Room 205

Santa Barbara, CA 93195

Instead of calling Exec, I suggest creating a Java Native Interface to

R.  The Ptolemy/Matlab interface uses JNI.  

The advantage to a JNI interface is that you can have tighter coupling

with R or Matlab

The Java Exec interface is tricky to use, since command line parsing

gets tricky, and the semantics of reading and writing to a process can

be strange.  You could use files to tranfer data to and from R, but

this can get tricky as well.  

In Ptolemy Classic, we had some common infrastructure that allowed

us to interface to Matlab and Mathematica.  Perhaps we could build

on the Matlab interface?

There is also code in $PTII/jni that makes it fairly easy to wrap

C functions.  Right now, the UI is broken, but the backend works.

I wrote an Exec actor in Ptolemy II 4.0, and frankly, I'm not proud of

it. It does the minimal, it executes a sub process and gets the result.

I tried hacking around with a better actor that was more complex, but

ran in to problems.  It seems like using PN to read and write data is

probably the way to go.  I was disappointed that I could not implement

an interactive shell.  I was surprised that Unix pipes, something I've

trivially used forever, is a little tricky to implement in sdf.

I'm not particularly wedded to this version of the Exec actor, 

in the end, we decided to have something simple.

On Feb 25, 2004, I wrote the following to the ptolemy internal mailing

list.

>> I simplified the Exec actor so that fire() does not return until 

>> the subprocess returns.  

>>  

>> This change makes it impossible to use this actor to invoke bash once

>> and send different commands to the same bash process repeatidly -

>> so, we cannot use the InteractiveShell to invoke commands in

>> the same running bash process, we would need to start a separate

>> process for each command.  To use InteractiveShell, we would need to

>> have some sort of PN specific version of Exec (which I did not

>> implement).

>>  

>> I left the command parameter as a PortParameter so that the

>> I can invoke the actor over an over with a different command line.

>> This makes it possible to write "Run all demos" as a model.

>> There is such a model in the test suite.

>> 

>> Now, if the subprocess terminates with a non-zero value, fire() throws

>> an exception.

>> 

>> Also, I made the environment parameter be a record, though this change

>> does make the code more complex.

>> 

>> Also, I looked into sending and end-of-transmission (EOT) character

>> as a way of terminating a cat process.  EOT is Control-D or \04.

>> I can send a \04 character, but cat does not seem to notice

>> and it does not terminate.

>> 

>> The following example illustrates the problem.

>> 

>> I create a text file that has an embedded Control-D char and

>> run it through cat and cat does not care.  

>> 

>> cxh@maury 85% cat controld.c

>> #include <stdio.h>

>> int main(int argc, char ** argv) {

>>     printf("%s%c%s", "foo", 4, "bar");

>> }

>> cxh@maury 86% cc controld.c

>> cxh@maury 87% a.out > controld.txt

>> cxh@maury 88% od -c controld.txt

>> 0000000   f   o   o 004   b   a   r

>> 0000007

>> cxh@maury 89% cat <controld.txt

>> foobarcxh@maury 90% 

>> 

>> I think Control-D handling is done in the shell.

>> 

>> I could modify Exec so that when it sees a Control-D it terminates the

>> process, but this would terminate the subprocess too soon if the

>> process was very long running.  

-Christopher