Event server twiki page

There is now an ATLAS twiki page for the event server scheme described in the earlier post by Tadashi
https://twiki.cern.ch/twiki/bin/viewauth/Atlas/EventServer
  - Torre

Kick-off meeting for managing the Analysis applications in DEFT, and the Web interface for that process

On June 13th, 2013 there was a meeting at CERN during which we discussed the initial requirements and parameters of the project aimed at supporting the analysis workflows in Prodsys2.

Present: J.Elmsheuser, N.Ozturk, M.Potekhin, A.Stradling

The scope of the items presented and discussed was as follows:

• description of the updated analysis model which involves "slimming" and "skimming"
• requirement for the user interface that would be optimal to support this particular mode of processing
• discussion of whether the database schemas being developed in Prodsys2 in the context of managed production can be extended and reused to cover the use cases presented
• itemization of purely technical issues that already are on our plate and which will need to be resolved very soon
• exploration of security, auth/auth and access policies
• characterization of the data elements present in the analysis stream as being similar to what's used in production, i.e. essentially relying on same dataset infrastructure and nomenclature
• evaluation of Django as a candidate platform for the Web service, based on the experience of the project participants
• usefulness of the XML format adopted for Meta-Task description in DEF
• the urgency of setting up a dedicated machine, to cover the needs of the project at CERN
• general timeline of the project

Consensus was reached and plans were made accordingly, in particular:

• the timeline of this project closely matches what was planned for "vanilla" DEFT commissioning
•  the starting point for development, in terms of the platform, will be Django
• authentication and access policies will be implemented by mapping identities obtained from CERN SSO and the encrypted DN from X509 certificate, to the user table
• the UI will provide ready capabilities for using templates and reusing typical tasks, with providing easy to use templates and settings based on the working group attribution (context-sensitive autoimation)

Separately, it was decided that the software development team will consist of the following personnel:

• M.Potekhin (project lead)
• A.Vaniachine (conceptual design, commissioning and QA)
• D.Golubkov (Web service design and coding)
• A.Stradling (technical design + module development)
• L.Sargsyan (Web service design and coding)
• S.Baranov (System Administrator, tech support and issue tracking, commissioning and QA)

The immediate deliverables were agreed upon:

• a Python module encapsulating the dataset naming logic according to the official nomenclature. To be done by A.Stradling, ETA end of June
• setting up a Web server with all components to support Django and a proper Apache configuration. To be done by S.Baranov and M.Potekhin, ETA June 22nd
• Having a service running to enable port scans etc, M.Potekhin, ETA end of June
• Prototype of a dataset registration service (functionality still not factored out of AKTR), M.Potekhin, ETA mid-July
• Updating DEFT schemas to support the Analysis workflow, D.Golubkov, ETA mid-July

According to the personnel breakdown presented above, the bulk of the Wev development will be done by D.Golubkov and L.Sargsyan.

June 18th update:
Laura and Dmitry started work on the initial task display module for the UI
July 3rd update:
Django prototype ready (with simplified schemas) for Tasks and Meta-Tasks
Dev server running, Apache TBD

April-May 2013: ProdSys II progress report (Maxim)

• Documentation work:
• Cleanup and updates of documentation (ADC page, blog, TWiki)
• Created most of the ProdSys TDR as well as subsequent updates. Presented at the Technical Interchange Meeting.
• Finalized the TDR based on additional materials sent in (Alexei, Kaushik, Torre)
• Major edits to PanDA LDRD
• Design and Development:
• "Communication" table created to serve as the command passing medium between DEFT and JEDI
• "Task parameter" object added as a CLOB
• Core DEFT coding:
• instrumented deft-cli to traverse the meta-task graph and process it repeatedly, in order to provide a ready and reproducible way to benchmark and measure performance.
• add the "task parameters" capability, a necessary feature. Translates into a CLOB in Oracle. JSON format chosen. Documentation updated.

• JEDI-alpha:
• Communicated with Tadashi and Dmitry to ensure consistency in schemas and APIs while working on DEFT and JEDI-alpha
• Improved documentation for DEFT/JEDI Interaction

• "DEFT-alpha" (codename):
• as the next iteration of the integration test, and based on the successful test of JEDI-alpha, implementing data entry in DEFT database to be injected into JEDI for "standard" processing

• Testing:
• In a first performance test, a simple 3-stage meta-task traversal (w/o DDM or any other such external system interaction) yielded these numbers: 9 seconds of elapsed time on an interactive node, to process 1000 meta-tasks. Conclusion - the performance of the core DEFT logic won't be the limiting factor in determining scalability of the system, as the likely latencies due to graph traversal seem to be acceptable. Keep in mind that the graph traversal processes can be easily farmed, to provide sufficient scaling.

• Common Analysis Platform:
• Participated in the CAP Meeting on 5/2/2013

DEFT/JEDI Communication Redux

General Notes on Communication

There has been progress in the design of the JEDI database schemas, documented in a separate section of the JEDI Twiki. Among a few other detail, there is a "COMMAND" column in the task table. This is a reminder that the DB is acting as the point of interaction and effectively an asynchronous messaging medium for DEFT and JEDI. Both are allowed to post requests to each other. Human operators are also capable to post requests to either of these systems, under certain conditions.

We reiterate what was previously stated with regards to DEFT/JEDI interaction:

• both components periodically do a database sweep, i.e. the operation is 100% asynchronous and there are no in-memory processes that are bound to a specific Panda Task or job
• the database is the medium for DEFT/JEDI communication
• JEDI never creates or deletes tasks or modifies the meta-task topology otherwise. It can post a request to DEFT to get this done. Thus, the important functionality remains withing one unit of code and is manageable. This sounds simple but it's fundamental for the system viability.
• examples for the previous item in action involve live task augmentation, and a related subject of handling the missing file issue. In the latter case, an additional task is added to the Meta-Task to make up for the lost statistics. Such request is formulated by JEDI, picked up by DEFT and is translated into a task, which is in turned picked up by JEDI.

After some consideration, we arrived to a solution which has a separate table to store the semaphores/commands

Task Parameters

Task parameters are a necessary attribute of the task object. They are essentially schema-free tuples of strings (could have a different implementation, but that's the reality). We will use CLOB to store these in Oracle DB, probably in a separate table to not impact performance.

In the context of JEDI-alpha, a good choice (agreed by most) is using JSON as the format for storage and inter-process communication.

Curious case of related datasets

Going through the use cases for DEFT/JEDI, I have noticed that there is a feature implicit in how the current workflow works, and not well understood by many people (including myself). It's the mechanisms that "bundles together" the datasets that are related in the processing logic, i.e. a PanDA task may produce datasets B,C based on an input dataset A, and datasets D,E based on another dataset F. Since there is no concept of Meta-Task in ProdSys I/PanDA, the inter-task chains of dependencies among the datasets are maintained semi-manually. The situation is represented by the simplified diagram below:

In this diagram, different colors represent different logical connection between datasets, for example DS4 is produced by task T3 to result in creation of the dataset DS7 (also colored red). Same applies to "blue" datasets in the same diagram. For completeness' sake, we also have the dataset DS5 there, which is thought to be the result of processing both "blue" and "red" input datasets.

At first glance, this presents a complication for the graph model employed in ProdSys II, since it introduces dependencies involving datasets, already modeled as edges in the Meta-Task graph. One approach would be to create a conjugate graph where the edges (datasets) become nodes and their connections (tasks) would be edges (the conjugate graph is also know as line graph, in graph theory). Then there is an option to store such graph in a separate GraphML file, or store it in the same file which will result in two disconnected graphs created upon parsing.

Either solution creates its own set of problems, and both share the necessity for crafty logic which is required to keep referential integrity when manipulating Meta-Tasks graphs, in operations like template processing etc.

Alternatively, one may try to implement an equivalent of a "port" in the task node. Unfortunately, this feature is not supported in the parser which comes with NetworkX. While it's always possible to roll our own, this likely negates one of the advantages of using this package.

A possible solution to this dilemma is to go back to the basic definition of the term "task", which originally was thought of as a unit of processing with one dataset coming in, and one dataset being output. If we lift the restriction on the number of datasets in that definition but introduce  "conservation of color", i.e. define the "task" as a processing unit which only operates on related "sets of datasets", we end up with essentially a partitioned but connected graph.

This actually amounts to a relatively straightforward reformulation of the graph presented above, with creation of additional nodes. For example, the picture above is transformed into the following:

 The new feature in this graph is that different tasks can have dependencies on same group of datasets, such as DS2 and DS3. What allows this to work in practice is that while the edges representing these different part of the graph are distinct in the graph description, they refer to the same datasets whose state is managed by JEDI.

Note that the "prime" tasks in this diagram share all or most of their attributes with the original ones, and only differ by the input and output, however this is taken care of in the graph itself.

February-March 2013: ProdSys II progress report (Maxim)

Documentation work:

• This blog: created tags "prodsys1" and "prodsys2" for better search capability.
• Created a common navigation header (bar) that can be included in all ProdSys TWiki pages.
• References to DEFT and further details added to documentation on the ProdSys pages.
• Corrections in DEFT/JEDI interface description as per Tadashi's comments.
• Prepared Abstract for the ProdSys paper (CHEP). Abstract approved by ATLAS and submitted.
• Presentation for the CMS/ATLAS Common Analysis Platform on 2/28/2013:
• Based on the announcement on 2/14 of a CMS development largely parallel to what we do in ATLAS
• Potential redundancy, under-utilization of PanDA capability, suboptimal database load
• Clear potential for common development and platform

• Presentation for the ATLAS Software and Computing Workshop, March 11-15 2013
• Meeting with Wolfgang to discuss progress and requirements

Development:

• DEFT prototype: functionality complete
• SVN project created, code checked in
• Continuous updates and checkpoints
• Naming of the SVN tree as per Tadashi's comments
• Tested database schemas for the Task, Dataset and Meta-Task objects.
• Extensive refactoring and rewrite of the main code unit due to lots of new functionality and increased complexity, the application has become a simple CLI driver for underlying classes.
• Dedicated test of the code state-switching functionality
• Improvements in logging functionality, Logger class created based on standard Python package
• Started work on the Dependency Model for datasets

Deft v1.0 - the prototype - is in SVN now

The prototype of the Deft system is now in a stage where it has a minimal but complete set of functionality as a standalone system. Integration with Jedi/PanDA will be achieved in a future version. The v1.0 designation should not be taken too seriously, since it's only significance is that we have a working baseline prototype, and not a pre-production version.

The cornerstone of the Deft design is its reliance on the graph model to handle Meta-Tasks and their components. This approach is prevalent in computer science and industry and there is no good reason to not follow it. In addition, an effort was made to use existing and proven software components in order to minimize the amount of application-specific code and significantly improve maintainability. Specifically, we use NetworkX as the graph engine, and the Workflow component of PyUtilib as the state machine.

Right now, Deft  has the following capabilities:

• Parsing Meta-Task definition supplied by the user in industry-standard GraphML format. The latter is a portable XML schema, developed for graph description and modeling, and supported in a number of current applications which can be used to visualize, explore and edit the meta-task structure.
• Building Meta-Tasks based on templates, and likewise clone Meta-Tasks based on previously existing GraphML descriptions. This was one of the crucial requirements put forth by Wolfgang. 
• Analysis of the Meta-Task and state transitions of its components, i.e. individual tasks.
• Tasks can be set to "armed" or "disarmed" state which will either enable or disable automated transition to the next state during processing by Deft.
• Integration with Oracle: implementation of the Adjacency Table technique to store Graph information in RDBMS.
• Interoperability between GraphML (XML) and RDBMS storage, whereby the data can be sourced from, and written to either.

The DEFT suite of Python modules and supporting materials is now in CERN SVN .

Gephi - graph vizualization and manipulation software

Gephi is a popular and powerful software tool which allows the user to create, import, manipulate, and visualize graphs of considerable complexity. All graph data can be easily manipulated. Gephi is capable of sophisticated analysis of graph characteristics. Import and export can be done in a wide variety of formats, which importantly include GraphML and a couple of other most popular formats. There is also CSV import/export option, which allows the user to archive and edit data in spreadsheet and other applications, as well as perform database import/export.

Gephi is described in a Wikipedia article. The project is sponsored by Google.

From initial experience, it appears that Gephi has the potential to fill the role of the graphic editor of meta-task requests. It can serve as the basis for creation and maintenance of meta-tasks templates. Another interesting possibility is to use Gephi as a turnkey meta-task monitoring tool, which only requires periodic dumps of meta-task data from Deft to operate.

Gephi encompasses a rich graph analytics toolkit. The screenshots below only illustrate the most basic functions of Gephi, which is graph visualization and graph data manipulation.They also demonstrate flexibility in choosing characteristics of the graph rendering.

Graph Visualization in Gephi (click to enlarge)

Data Editing Window (click to enlarge)

Different visualization style (click to enlarge)

Deft and Jedi

Design Parameters

In the initial design, Jedi was designated as the component most closely coupled with the PanDA service itself, while Deft was considered a more platform-neutral database engine for bookkeeping of tasks (hence its name). Assuming the decision on such separation is made,  it makes sense to leverage this factorization to take advantage of component-based architecture. In the following we assume that Deft will have a minimum level of dependency on PanDA specifics, while Jedi takes advantage of full integration with PanDA. Wherever possible, Jedi will have custody of the content stored in the database tables, and Deft will largely use references to these data.

Communication

The complete logic of the interaction of Deft with Jedi (and vice versa) is being worked out and will be documented on the ProdSys TWiki when it's ready. For now, let's note a few features of the Deft/Jedi tandem design:

• in managing the workflow in Deft, it unnecessary (and in fact undesirable) to implement subprocesses and/or multi-threaded execution such as one in PaPy etc. This is because Deft is NOT a processing engine, or even a resource provisioning engine, as is the case in some Python-based workflow engines. It's a state machine that does not necessarily need to work in real time (or even near time).
• there are a few ways to establish communication between Deft and Jedi, which are not mutually exclusive. For example, there may be callbacks, and database token updates, which may in fact co-exist. If a message is missed for some reason, the information can be picked up during a periodic sweep of the database.

An interesting question is whether the database should be exposed through the typical access library API, or as a Web Service. The latter provides much better flexibility at the cost of potential reduction in performance. Needs to be evaluated.

The Job database in Jedi will experience a significant read/write load. It's important to insulate Deft with its task databases from excessive read/write activity. This can be accomplished in a number of ways, for example the number of jobs finished in a task can be updated periodically but not as frequently as the actual job completion rate.

Data Services Interaction

Design documentation for Jedi is presented on a CERN TWiki page .

Questions/comments:

• in the "Workflow" section, the dataset name generated automatically. Does this part belong in Jedi? Presumably yes, or it can be handled by a separate service
• Deft operates by traversing its graph model of the workflow, taking action at each node which depends on the node's incoming edges. It is therefore possible that operations such as insertion into the dataset table are done by calling a method in a plug-in or calling a web service. If the latter is implemented, it becomes a separate component complementary to Deft/Jedi.
• Incremental task execution: currently the proposed solution is the same on both Deft and Jedi side.  See the "Augmentation" section of the ProdSys Workflow page, and Incremental Task section of the Jedi page. Basically, the state of relevant task nodes is reset, and new edges (datasets) added. The issue of what to do with tasks that already transitioned to the Archived state can be resolved by Meta-Task cloning. Specifically, if any of the tasks went to "archived", the whole Meta-Task must go to "archived" state and further changes are disallowed, but cloning is allowed.
• General note: Deft does not contain any reference to individual files. One consequence of this is that "open dataset handling" is done in Jedi. It might be a good idea to avoid this mode of operation, if possible.

Performance study of the event table for JeDi

The event table is a new table for JeDi which keeps track of progress of job at event level. We are planning to use the table for the even server and event-level job splitting. Here is the first result of the performance test for the event table. The table was created in INTR with the following schema:

Name	Type
PANDAID	NOT NULL NUMBER(11)
FILEID	NOT NULL NUMBER(11)
JOB_PROCESSID	NOT NULL NUMBER(10)
DEF_MIN_EVENTID	NUMBER(10)
DEF_MAX_EVENTID	NUMBER(10)
PROCESSED_UPTO_EVENTID	NUMBER(10)

where PANDAID and FILEID are IDs in job and file tables, JOB_PROCESSID is ID for a subprocess, DEF_MIN_EVENTID and DEF_MAX_EVENTID define the range of events for the subprocess, and PROCESSED_UPTO_EVENTID represents how many events are
done so far. The primary key is the combination of PANDAID, FILEID, and JOB_PROCESSID. The table physical layout is range-partitioned based on PandaID. The table is index-organized but also partitioned, which is handy for avoiding row-by-row deletion and tree fragmentation. Now each partition will fit 1 million PandaIDs.

The idea of the event server is shown in Fig.1.

Figure 1. A schematic view of the event server

In the event server scheme, multiple pilots process the same job and file in parallel, but each of them takes care of only a different range of events. When the panda server receives a request from a pilot, the panda server sends a range of events (e.g., DEF_MIN_EVENTID=0 and DEF_MAX_EVENTID=99) to the pilot together with job specification and one record is inserted to the event table. The pilot sends heartbeat at every N events processed, so that PROCESSED_UPTO_EVENTID of the record is updated in the event table. When another pilot comes, the panda server scans the event table, and sends a new range of events (e.g., DEF_MIN_EVENTID=100 and DEF_MAX_EVENTID=299) to the pilot if there are events remaining for the job and file.

A script was implemented to emulate interactions between the panda server and the database. The script spawned 5000 child processes and 1000 jobs were processed in parallel, i.e., 5 child processes were used for one job. Each child process sends heartbeat every 2 sec. The script processed roughly 0.4 million jobs per day, which corresponds to the half of the number of jobs processed per day in the current system. Note that INTR is hosted by a low performance machine since it is a testbed and not all jobs will use the event server scheme. Although the result might be acceptable, we will continue stress tests to see if further optimization is possible.

Notes on Workflow Management

Workflow management is a rich topic in both theoretical and practical sense. There are a large number of concepts and software products which support WM. There is specialization according to the domain, such as managing business workflow vs scientific workflow. There are some shared commonalities across these boundaries, e.g. widely adopted use of XML as a means to describe or capture the state of the system. However, business-oriented WMS do not appear well suited for computational workflow management. We note YAWL and BEPL as commercial oriented systems, while Kepler was designed to drive scientific workflow applications.

---

Paper on workflow scheduling on the Grid.

---

There is a useful Paper on Workflow Patterns. Illustrations below represent two of workflow patterns of interest in our application, among others.

Introduction to ATLAS PanDA Production System

October 23, 2012

The Role of ProdSys

The natural unit of workload that is handled by PanDA is a single payload job. Defining the exact nature of the payload, source and destination of data and various other parameters that characterize a job is outside of the scope of core PanDA itself.

ATLAS Production System serves an extremely important role of defining jobs for a large part of the workload handled by PanDA. Jobs are defined in large sets that constitute "tasks", and are formulated to fulfill "task requests". Each task has a number of attributes, set in accordance with a particular request. Each task is typically translated into a large number of jobs. The existing Production System consists of a task request interface, a set of scripts that translate tasks into respective jobs, and a few tools for modification of certain parameters of active tasks.

Individual job definitions in the existing system are created based on the task parameters and remain static for the duration of the task execution. Data pertaining to requests, tasks and jobs reside in the database, and operation of the Production System can be described as transforming one object into another, starting with requirements, formulating tasks and then creating a list of jobs for each task, for execution in PanDA.

---

Motivations for system evolution

Motivations for evolving the ATLAS production system come from realization that we need to address the following:

• The concept of Meta-Task. Absent in the original product (ProdSys I), it emerged based on operational experience with PanDA and its workflow. It became the central  object in the workflow management and must be properly introduced into the system.
• Operator intervention and Meta-Task recovery: there must be adequate opportunities for the operators and managers to direct the Meta-Task processing, be able to start certain steps before others are defined, augment a task, and recover from failures in an optimal way.
• Flexibility of job definition (e.g. making it dynamic as opposed to static once the task is created): there are a number of advantages that we hope can be realized once there is a capability to define jobs dynamically, based on the resources and other conditions present once the task moves into the execution stage
• Maintainability: the code of the existing Production System was written "organically", to actively support emerging requests from users, and starts showing its age
• Scalability: there are issues with the way the interaction between the ProdSys software and the database back-end, which lead to lockup condition of the database when a transaction is handled, and also the issue of general insufficient throughput when inserting tasks and other data into the system
• Ease of use: there is currently a great amount of detail that the end user (Physics Coordination) must define in order to achieve a valid task request. It's desirable to automate the task creation process, whereby cumbersome logic is handled within the application, and the user interface is more concise and transparent.