December 2012: ProdSys II Progress Report (Maxim)

12/16/12 to 12/31/12

Documentation work:

• Quick links added to the top of all core pages in the ProdSys TWiki
• Added "Meta-Task Recovery" to the Requirements and to the Workflow pages, to reflect the previous documented requirements for the system. This dates back to February 2012, and has been mentioned elsewhere.

Development:

• Evaluation of Python packages for Workflow Management:
• Soma workflow
• Weaver workflow
• Other items as per TWiki documentation

• Evaluation: Graph representation in XML, standard solutions studied in conjunction with parsing tools
• Preliminary selection of GraphML as the language enjoying standardization, fairly simple sytax and parsing support
• A prototype of the graph builder and workflow engine created, based on:
• "GraphML" as the input language
• "Networkx" as the serializer/deserializer
• "PyUtilib" as the workflow engine

12/01/12 to 12/15/12

Meetings:

• BNL
• a meeting was held at BNL to discuss the current status of the ProdSys documentation and the initial design of the new system. Present: K.De, V.Fine, A.Klimentov, S.Panitkin, M.Potekhin, T.Wenaus
• the Graph Model was reviewed and approved
• decision made to avoid heavy-weight XML-based solutions

• FNAL/CMS
• a Common Platform Meeting was held at FNAL (Dec.5-7). M.Potekhin attending in person,  and T.Wenaus remotely.
• a review of the work previously done under the mandate of Feasibility Study of PanDA/GlideinWMS Integration. Fairly detailed discussion of scalability issues.
• a brief overview of the current work on ProdSys II. Burt has suggested looking into DagMan application (note: rejected after documentation review).

• FNAL/LBNE
• Initial meeting at FNAL with LBNE team, some ProdSys ideas discussed, among other issues pertaining to the general software management in LBNE.

• FNAL/BNL
• Initial meeting at BNL with the local LBNE team. Corroborated on FNAL discussion, agreed on basic principles of software coordination
• A draft document created describing the principles of Software Coordination at LBNE

Documentation work:

• Workflow page  has been created
• Considerable cleanup of TWiki pages for Panda, ProdSys, TaskModel
• Some presentation material added
• Added "Augmenting Live Meta-Task" and "Partially Defined Meta-Task" use cases
• Permanent links added to this Blog (front page)

Development:

• Evaluation of Python packages for Workflow Management:
• pyutilib.workflow
• romanchyla/workflow on Github