Infrastructure Design
Critical API Functionality
- Job Definition
- Set of Tasks
- Set of input data - would it make sense to specify this
- Set of required executables - for our project, should we require that
all the code for the run reside in a single .jar file?
- Task Definition
- Communication Channel for Task (TaskContext in Parabon)
- Parameter list, data dependencies, executable dependencies
for Task (TaskSpec in Parabon)
- Job Submission
- Monitoring Jobs/Tasks
- User application should be able to "listen" to the jobs and tasks.
The "Listener" design pattern makes sense in this context (Parabon uses this
pattern).
- Communication
- "Master" node communication of results, checkpoints, progress, status,
and statistics to the user node.
- "Volunteer" node communication of results, checkpoints, progress, status,
and statistics to the "master" node.
- Inter-task communication? This does not have to be implemented as
direct communication between tasks running on different nodes, but perhaps
the API should be designed with that possibility in mind.
Daemon Design
There are at least 2 daemons to implement:
- "Master" node daemon - waits for job submission and then dispatches
tasks to nodes.
- "Volunteer" node daemon - waits for tasks and data to arrive and then
runs the tasks.
Both daemons also have to handle monitor requests and the communication
results, statistics, etc.