Deft and Jedi

Design Parameters

In the initial design, Jedi was designated as the component most closely coupled with the PanDA service itself, while Deft was considered a more platform-neutral database engine for bookkeeping of tasks (hence its name). Assuming the decision on such separation is made,  it makes sense to leverage this factorization to take advantage of component-based architecture. In the following we assume that Deft will have a minimum level of dependency on PanDA specifics, while Jedi takes advantage of full integration with PanDA. Wherever possible, Jedi will have custody of the content stored in the database tables, and Deft will largely use references to these data.

Communication

The complete logic of the interaction of Deft with Jedi (and vice versa) is being worked out and will be documented on the ProdSys TWiki when it's ready. For now, let's note a few features of the Deft/Jedi tandem design:

• in managing the workflow in Deft, it unnecessary (and in fact undesirable) to implement subprocesses and/or multi-threaded execution such as one in PaPy etc. This is because Deft is NOT a processing engine, or even a resource provisioning engine, as is the case in some Python-based workflow engines. It's a state machine that does not necessarily need to work in real time (or even near time).
• there are a few ways to establish communication between Deft and Jedi, which are not mutually exclusive. For example, there may be callbacks, and database token updates, which may in fact co-exist. If a message is missed for some reason, the information can be picked up during a periodic sweep of the database.

An interesting question is whether the database should be exposed through the typical access library API, or as a Web Service. The latter provides much better flexibility at the cost of potential reduction in performance. Needs to be evaluated.

The Job database in Jedi will experience a significant read/write load. It's important to insulate Deft with its task databases from excessive read/write activity. This can be accomplished in a number of ways, for example the number of jobs finished in a task can be updated periodically but not as frequently as the actual job completion rate.

Data Services Interaction

Design documentation for Jedi is presented on a CERN TWiki page .

Questions/comments:

• in the "Workflow" section, the dataset name generated automatically. Does this part belong in Jedi? Presumably yes, or it can be handled by a separate service
• Deft operates by traversing its graph model of the workflow, taking action at each node which depends on the node's incoming edges. It is therefore possible that operations such as insertion into the dataset table are done by calling a method in a plug-in or calling a web service. If the latter is implemented, it becomes a separate component complementary to Deft/Jedi.
• Incremental task execution: currently the proposed solution is the same on both Deft and Jedi side.  See the "Augmentation" section of the ProdSys Workflow page, and Incremental Task section of the Jedi page. Basically, the state of relevant task nodes is reset, and new edges (datasets) added. The issue of what to do with tasks that already transitioned to the Archived state can be resolved by Meta-Task cloning. Specifically, if any of the tasks went to "archived", the whole Meta-Task must go to "archived" state and further changes are disallowed, but cloning is allowed.
• General note: Deft does not contain any reference to individual files. One consequence of this is that "open dataset handling" is done in Jedi. It might be a good idea to avoid this mode of operation, if possible.