ATLAS Production System Twiki

Join this site and follow this blog to be up to date, or simply subscribe to our RSS feed!

Permanent Documentation Links
Blog Tags: prodsys1, prodsys2

Monday, January 28, 2013

Deft v1.0 - the prototype - is in SVN now

The prototype of the Deft system is now in a stage where it has a minimal but complete set of functionality as a standalone system. Integration with Jedi/PanDA will be achieved in a future version. The v1.0 designation should not be taken too seriously, since it's only significance is that we have a working baseline prototype, and not a pre-production version.

The cornerstone of the Deft design is its reliance on the graph model to handle Meta-Tasks and their components. This approach is prevalent in computer science and industry and there is no good reason to not follow it. In addition, an effort was made to use existing and proven software components in order to minimize the amount of application-specific code and significantly improve maintainability. Specifically, we use NetworkX as the graph engine, and the Workflow component of PyUtilib as the state machine.

Right now, Deft  has the following capabilities:
  • Parsing Meta-Task definition supplied by the user in industry-standard GraphML format. The latter is a portable XML schema, developed for graph description and modeling, and supported in a number of current applications which can be used to visualize, explore and edit the meta-task structure.
  • Building Meta-Tasks based on templates, and likewise clone Meta-Tasks based on previously existing GraphML descriptions. This was one of the crucial requirements put forth by Wolfgang. 
  • Analysis of the Meta-Task and state transitions of its components, i.e. individual tasks.
  • Tasks can be set to "armed" or "disarmed" state which will either enable or disable automated transition to the next state during processing by Deft.
  • Integration with Oracle: implementation of the Adjacency Table technique to store Graph information in RDBMS.
  • Interoperability between GraphML (XML) and RDBMS storage, whereby the data can be sourced from, and written to either.
The DEFT suite of Python modules and supporting materials is now in CERN SVN .

No comments:

Post a Comment