Deft v1.0 - the prototype - is in SVN now

The prototype of the Deft system is now in a stage where it has a minimal but complete set of functionality as a standalone system. Integration with Jedi/PanDA will be achieved in a future version. The v1.0 designation should not be taken too seriously, since it's only significance is that we have a working baseline prototype, and not a pre-production version.

The cornerstone of the Deft design is its reliance on the graph model to handle Meta-Tasks and their components. This approach is prevalent in computer science and industry and there is no good reason to not follow it. In addition, an effort was made to use existing and proven software components in order to minimize the amount of application-specific code and significantly improve maintainability. Specifically, we use NetworkX as the graph engine, and the Workflow component of PyUtilib as the state machine.

Right now, Deft  has the following capabilities:

• Parsing Meta-Task definition supplied by the user in industry-standard GraphML format. The latter is a portable XML schema, developed for graph description and modeling, and supported in a number of current applications which can be used to visualize, explore and edit the meta-task structure.
• Building Meta-Tasks based on templates, and likewise clone Meta-Tasks based on previously existing GraphML descriptions. This was one of the crucial requirements put forth by Wolfgang. 
• Analysis of the Meta-Task and state transitions of its components, i.e. individual tasks.
• Tasks can be set to "armed" or "disarmed" state which will either enable or disable automated transition to the next state during processing by Deft.
• Integration with Oracle: implementation of the Adjacency Table technique to store Graph information in RDBMS.
• Interoperability between GraphML (XML) and RDBMS storage, whereby the data can be sourced from, and written to either.

The DEFT suite of Python modules and supporting materials is now in CERN SVN .

Gephi - graph vizualization and manipulation software

Gephi is a popular and powerful software tool which allows the user to create, import, manipulate, and visualize graphs of considerable complexity. All graph data can be easily manipulated. Gephi is capable of sophisticated analysis of graph characteristics. Import and export can be done in a wide variety of formats, which importantly include GraphML and a couple of other most popular formats. There is also CSV import/export option, which allows the user to archive and edit data in spreadsheet and other applications, as well as perform database import/export.

Gephi is described in a Wikipedia article. The project is sponsored by Google.

From initial experience, it appears that Gephi has the potential to fill the role of the graphic editor of meta-task requests. It can serve as the basis for creation and maintenance of meta-tasks templates. Another interesting possibility is to use Gephi as a turnkey meta-task monitoring tool, which only requires periodic dumps of meta-task data from Deft to operate.

Gephi encompasses a rich graph analytics toolkit. The screenshots below only illustrate the most basic functions of Gephi, which is graph visualization and graph data manipulation.They also demonstrate flexibility in choosing characteristics of the graph rendering.

Graph Visualization in Gephi (click to enlarge)

Data Editing Window (click to enlarge)

Different visualization style (click to enlarge)

January 2013: ProdSys II Progress Report (Maxim)

01/10/13 to 01/31/13

Documentation work:

• Updates in the general PanDA TWiki pages
• References to Jedi documentation on the ProdSys pages
• Additions to Workflow page based on recent experience

Development:

• Deft prototype development in progress with code cleanup and proper logging added
• Implemented XML file-based meta-task database as a testing and integration tool
• Implemented and tested the meta-task template functionality as defined in the requirements
• Set-up of Oracle software at BNL to move towards DB integration with JEDI

Graph processing:

• Installation of, and experimentation with, the following tools: graphviz and Gephi
• Conclusion: adequate graph editing and visualization capabilities of Gephi, good enough to actually edit meta-tasks
• R&D with Wireit and jsPlumb packages as tools for visualizing and editing meta-tasks and tasks, in a Web client such as browser

 Misc:

• LBNE Documentation Review. "Redmine" pages, SVN integration