Plans for JEDI-α

Note posted in June 2013: since this post was originated and updated, the activity which it describes led to a successful commissioning of JEDI-α. Work is still under way to tune the database schemas and to complete functional testing.


This is a schematic description of flow of logic JEDI-α and how it compares to Vanilla ProdSys I:

Sequence:
 

Vanilla	[1] "Task Definition"	AKTR tables	[2] ProdSys tables	[3] ProdSys1 sequence
α	-	-	[2'] DEfT tables	[3'] JEDI sequence

Description:

• [1] In the Task Definition object a flag will be added, to differentiate ProdSys1/ProdSys2. If the flag is set to "ProdSys2" then DEfT tables will be filled by the system
• [2'] Tables (as defined in https://twiki.cern.ch/twiki/bin/viewauth/Atlas/WorkFlow) will be filled by Dmitry
• [3'] JEDI-alpha will take info from the above DEfT tables

Upon implementation, we will be able to debug ProdSys II w/o disturbing the current system and even run production with both systems if needed.

Please comment.

Question: who will register the output dataset(s) in [2']?
In principle it will be transparent for current datasets/containers registration.

Probably ProdSys2 tasks can have a special state and in this case it will be easily seen in monitoring.

Comment from MP: this could be a "type" or "origin" column in the task. Different "state" can make things more complex.

April-May 2013: ProdSys II progress report (Maxim)

• Documentation work:
• Cleanup and updates of documentation (ADC page, blog, TWiki)
• Created most of the ProdSys TDR as well as subsequent updates. Presented at the Technical Interchange Meeting.
• Finalized the TDR based on additional materials sent in (Alexei, Kaushik, Torre)
• Major edits to PanDA LDRD
• Design and Development:
• "Communication" table created to serve as the command passing medium between DEFT and JEDI
• "Task parameter" object added as a CLOB
• Core DEFT coding:
• instrumented deft-cli to traverse the meta-task graph and process it repeatedly, in order to provide a ready and reproducible way to benchmark and measure performance.
• add the "task parameters" capability, a necessary feature. Translates into a CLOB in Oracle. JSON format chosen. Documentation updated.

• JEDI-alpha:
• Communicated with Tadashi and Dmitry to ensure consistency in schemas and APIs while working on DEFT and JEDI-alpha
• Improved documentation for DEFT/JEDI Interaction

• "DEFT-alpha" (codename):
• as the next iteration of the integration test, and based on the successful test of JEDI-alpha, implementing data entry in DEFT database to be injected into JEDI for "standard" processing

• Testing:
• In a first performance test, a simple 3-stage meta-task traversal (w/o DDM or any other such external system interaction) yielded these numbers: 9 seconds of elapsed time on an interactive node, to process 1000 meta-tasks. Conclusion - the performance of the core DEFT logic won't be the limiting factor in determining scalability of the system, as the likely latencies due to graph traversal seem to be acceptable. Keep in mind that the graph traversal processes can be easily farmed, to provide sufficient scaling.

• Common Analysis Platform:
• Participated in the CAP Meeting on 5/2/2013