DEFT/JEDI Communication Redux

General Notes on Communication

There has been progress in the design of the JEDI database schemas, documented in a separate section of the JEDI Twiki. Among a few other detail, there is a "COMMAND" column in the task table. This is a reminder that the DB is acting as the point of interaction and effectively an asynchronous messaging medium for DEFT and JEDI. Both are allowed to post requests to each other. Human operators are also capable to post requests to either of these systems, under certain conditions.

We reiterate what was previously stated with regards to DEFT/JEDI interaction:

• both components periodically do a database sweep, i.e. the operation is 100% asynchronous and there are no in-memory processes that are bound to a specific Panda Task or job
• the database is the medium for DEFT/JEDI communication
• JEDI never creates or deletes tasks or modifies the meta-task topology otherwise. It can post a request to DEFT to get this done. Thus, the important functionality remains withing one unit of code and is manageable. This sounds simple but it's fundamental for the system viability.
• examples for the previous item in action involve live task augmentation, and a related subject of handling the missing file issue. In the latter case, an additional task is added to the Meta-Task to make up for the lost statistics. Such request is formulated by JEDI, picked up by DEFT and is translated into a task, which is in turned picked up by JEDI.

After some consideration, we arrived to a solution which has a separate table to store the semaphores/commands

Task Parameters

Task parameters are a necessary attribute of the task object. They are essentially schema-free tuples of strings (could have a different implementation, but that's the reality). We will use CLOB to store these in Oracle DB, probably in a separate table to not impact performance.

In the context of JEDI-alpha, a good choice (agreed by most) is using JSON as the format for storage and inter-process communication.