Executor - a Simple Directory/File-based Distributed Computing System

Looking for security information? Check here.
To get an overall view of how to use this software, please check the quick start guide or these nice images.
This software is released under the modified BSD license.

Examples

You'll find some example programs for the simple Java executor here.
AskTell.jar a simple Java demo that plays "guess a number" with you
hello.bat outputs hello followed by the program's parameters (if any)
hello.sh a similar program for Unix shells
salesmen.py the classic traveling salesman program with 11 cities. Can take many (10) minutes on lower end computers.
darts, an example for the R statistics package. This ZIP file contains both the data and the executable, but you can just upload it as File#1 in the Executor. Using a sample from https://www.dartmouth.edu/~chance/teaching_aids/data.html (originally Wall Street Journal), we calculate some summaries and indicate that in general trained experts trade a bit better than can be gained by throwing darts on the stock index.

What does Executor do?

Executor is a small but perfectly functional Java (1.5 or higher) JAR executable that is used to run programs in shared computer resources like a pool of computers that you and your friends own. Moreover, it can be used as a simple file sharing program: you can access the files of your home computer from your office computer.
Executor uses a shared or synchronized file system. The user creates a task in one computer (just by uploading a file) and another computer that has the required program will execute the task.
A class diagram of the program is here.

Who wrote it/uses it? What's the aim/status? Where can I get it?

I am node0, the program was written during my spare time 2013- and I use it daily. Mail: Executor-all@nongnu.org
I plan to incorporate changes and more functionality but I want to keep it small and simple.
You can download the software from savannah.

FAQ

Q: Why would I use Executor?
A: If you have a powerfull desktop computer with lots of installed programs, and a small laptop, you can launch your tasks using the laptop, they are executed in the desktop computer and the results delivered in the shared directory so you can recover them with the laptop.

Q: Why wouldn't I use something better like Condor, SGE, BOINC?
A: You should. But this one is really simple to set up and use.

Q: So, can I use Microsoft Office that is in my Windows computer by submitting a Word file by Executor in my Linux box?
A: Sadly, no. We can only manage batch type processing. Executor cannot transfer the GUI of MSoft Office into your Linux.

Q: What about security?
A: Optional, based on GnuPG. Please see here

Q: What is the protocol by which computing requests are submitted and executed?
A: The protocol is completely file-based. When a user submits a task by the submit GUI, Executor (1) creates a file with uniqueID.jd (jd for job description). When Executor (2) finds a jd file, it checks if it can execute it. If it can, it creates a file with the same uniqueID and extension "run". Data about execution (exec host, start time..) is written in this file. After the execution, the exit code is written in this file, too. Executor (3) can notify the user that the task has been executed, simply by observing the .run files.

Q: There is no scheduling?
A: Yes, very primitive. All computers write a .info file in the shared directory. This contains some characteristics (cpu, memory, supported programs) of the host. When a task arrives, any computer will check if they can run it. But if the host notices (by checking the files) that there is another more powerfull computer available for the task, it will wait for some time. If after the wait none has adopted the task, it will. This is why the scheduler class is called PoliteScheduler.

Q: How about interactive tasks, tasks that ask input from the user?
A: There is a simple (file-based, again) protocol called ASK-TELL. A program that wants input writes a question in its stdout, like this 

ASK what is your name?
Executor stdout notices this question and creates a pop-up in the host where the job was submitted. The program reads the answer in its stdin.
Please note that process management is a bit tricky. Your program must flush the stdout instantly after writing the ASK question, otherwise it will just wait forever. Moreover, when reading the reply, your program must be prepared to strip flow control characters \002 \004 from the reply.

Q: What is the syntax of the job description file?
A: Some of the attribute names can sound peculiar since they have been adopted from a more complicated program, but here they are:
clientid=xxxx.info defines the Executor who submitted the task
binaryinputfilename=filename the file that was given as File #1 by the submit GUI
workercommandline=command that the Executor will run
arguments=[optional] extra arguments from the submit GUI
datainputfilename=[optional] more input files

Q: Is there a Java API?
A: Yes, it is included in the same JAR file: 

* Submit a task such that the executable is in myFilename and myCmd is
* its command line.
public String submitTask(String myFileName, String myCmd, String taskName)

* API-type method: returns hosts/nodes that can execute your task.
* @param exTension extension like "sh" or a file name like "foo.sh"
* @param coreWeigth if true, return N*H for H where N is available cores.
* @return list of info file names
public String[] getIdsSuitableForTask(String exTension,
                                          boolean coreWeigth

Q: Is there a command line interface?
A: Yes, it is included in the same JAR file: 

-H filename: list hosts that can execute filename
-C filename: list cores that can execute filename
-S filename: submit task

Q: What new functionality is planned next?
A: If you look at a job description file, it actually looks like a sequence of variables. I have been thinking: could we support programming language constructs like FOR loops in these files, too? However, we would need to support this by the GUI, and it can become messy.