EasyGrid: a Grid Job submission system for High Energy Physics distributed analysis.

James Werner - jamwer2000@hotmail.com

Developing a grid based system for distributed analysis. This enables physicists at different institutes to use the computing resources available at various sites (including Tier 1 and Tier 2 centres). EasyGrid is an intermediate layer between user software and grid middleware. EasyGrid is responsible for job submission, copy executables at grid Storage Elements, recovery results and collect all information for post-mortem analysis (if something goes wrong). Users have to know nothing about grid to use EasyGrid and do their analysis. It suits users whose main interest is not in computer science but in High Energy Physics.

Project roadmap.

Seting grid site from scratch

These pages describe how to install grid LCG 2.6.0 software in a production site.
  1. LCG2 installation.
  2. User Interface.
  3. Information system.
  4. Resource Broker.
  5. Computer Element.
  6. Work node.
  7. Storage element.
  8. Testing the site.
  9. Operational procedures.
or you can follow a one web page procedure.

After install LCG software, grid manager needs to configure the site following Standards and configuration - making all transparent.

Installing BaBar software from scratch

These pages describe how to install BaBar software in a storage system (AFS or NFS).

The diagram shows the relationship between sites, and contains the links between the application, the configuration file and the physical data for each type of information. The path points to the directories and the configuration files. You will need this information to customise each package for your own environment.

  1. Basic infrastructure for Babar software.
  2. Babar requirements.
  3. Babar software installation.
  4. Babar software Configuration.
  5. Babar software Maintenance.
  6. Importing and checking new releases at new site.
  7. Downloading and configuring the Book Keeper database.
  8. Load local database with experiment data and monte carlo simulation
  9. Installing and configuring Configuration database - LIGTH VERSION.
  10. Installing and configuring Configuration database - FULL VERSION.
  11. Upgrading Condition and configuration database.
  12. Upgrading Book Keeper database.
  13. Upgrading Babar software.

Job Submission System - EasyGrid - S P E C I F I C A T I O N

The use of web pages to store specifications allows updates and immediate distribution for all users, as soon as new ideas and suggestions came from new tests and users. This section contains the available specification and describes how it was implemented in the prototype and in production, as well as how it will improve as project goes on. It will be part of users' manual as well, which makes its use more wide than a specification document only.

Important news: There are discussions about the final implementation for UK. See http://hepunx.rl.ac.uk/BaBar/uk-spgrid/babar-env-setup.html for more details. The meeting 2nd June gave new directions for the project:
--> EasyGrid will not optimise grid load, only provide a list of CE in the requirements clause. This function will be performed in the future by Resource Broker/RAL.
--> Instead use Glue Schema to publish site resources, will be use tags. RAL will update resources tags.
--> Datasets will be published by tags, with site resources and not more by RLS. RAL will update datasets tags.
--> dCache will be part of BookKeeping manager through some change in KanAccess.cfg. This is transparent to grid and babar software. Its implementation follows xroot approach and will be done by Bbk developers.

  1. Introduction.
  2. Requirements... under these circumstances.
  3. Methodology.
  4. Components and infrastructure.
  5. Job Submission System Modules.
  6. Job Submission Runtime.
  7. Job Submission System state-machine.
  8. Assessment and risk analysis.
  9. Application and Installation Standards conformity.
  10. Upgrade policies.
  11. Production support.
  12. Testing the environment.

How to program and run BaBar codes in grid.

These web pages describe each EasyGrid's functionalities:
  1. General HEP paradigm

  2. Requirements to use grid: paperwork and certificates.

  3. How to run Analysis BaBar Software: These pages contains instructions to run analysis programs with CM2. It contains the following main topics:

    At the end of the process (0.5 day work), you will be able to run analyses using release 14.5.2 and BaBar CM2 software.

  4. Programming BaBar Analysis Software.

    There are several ways to represent the information from the experiment, depending the user's knowledge of the reaction and the data available. The initial study uses histograms (mono and bi-parametric) to represent the number of events for some physical variable. Its possible to plot the number of particles with rest mass between 100MeV and 200MeV from the combination of 2 or more particles or gammas. Histograms are static because they do not allow a different study using their data. One can only plot different sections and using colors and superposition of histograms. Another possibility is to recover events with some specific characteristic, e.g. which could be selected through "cuts", and to generate ntuples. This is a more interesting way to store information, because as far as users know what events they want, it is possible to test several models from the selected events, with no need to search all the dataset. Each program is an independent approach for one specific problem. Probably, the user will need to join more than one of these programs to write a useful analysis software.

  5. Running your BaBar analysis software in Grid - easysub.

  6. Submitting root applications to grid - easyapp.

  7. Toolbox for Job Submission System: running batches, resub jobs, update datasets, ... everything else.

  8. EasyGrid production tar

Running BaBar CM2 Software with Grid - LCG2 at Manchester Grid TestBed

The grid available at Manchester University consists of 6 Worker Nodes, one Computer Element running, one resource broker, one information system, and one storage element running LCG2. The user interface is a remote computer. Data transfer is done through NFS. See the hardware configuration here.

  1. Script for Grid submission, status, and recovery.
  2. Running the tutorial example using grid.
  3. Running monte carlo using grid.

User manual for EasyGrid Prototype at Manchester's testbed.

  1. Initial startup and BaBar Analysis
  2. Submitting applications to grid using easyapp.
  3. Root source Submission.
  4. Monte Carlo Submission.
  5. Toolbox for Job Submission System.

Benchmark 1 - Particle Identification and Quality assurance: How to guarantee that everything (BaBar software and grid) is fine?

  1. Discriminate function for High Energy Particle Identification.
  2. Monte Carlo particle identification and EMC performance.
  3. Experimental data, Monte Carlo simulation and grid agreement.

Benchmark 2 - A complete HEP research project using Grid:

The project is a test analysis to assess the job submission prototype and to acquire information to write the specification for a production job submission system using grid infrastructure in a reliable way. It covers a range of branching rates, and physical results were assessed from previous experiments. Pi0 physics and decay is very well understood by QED, and is a common decay of tau leptons. This project will help newbies understand how to select events using cuts and selectors, and what criteria should be used in the selection process.

  1. Project description and overview.
  2. Software and cuts to select pi0 events.
  3. Running analysis software using grid.
  4. Algorithm 1: Results for any combination of gammas with VeryLoose selectors.
  5. Algorithm 2: Results for monoGAMMic gammas without mass minimisation with VeryLoose selectors.
  6. Algorithm 5: Results with VeryTight selectors and mass minimisation.
  7. References about parametric models:
  8. Analisys of Tau decaying in Pi+- Pi0 - M.Naisbit [pdf] [ppt]
  9. Hadronic structure in the decay tau to 1 pi0 (CLEO) [pdf]
  10. Structure functions in the decay tau to 2 pi0 (CLEO) [pdf]
  11. Branching rates and spectral functions of tau decays (ALEF) [pdf]
  12. The physics of hadronic tau decays [pdf]
  13. Papers in HEP

The results of this project were discussed in the BaBar Users Meeting on 9th February 2005, where I got users feedback to improve the specification. The total time to develop it was 4 weeks to show the proposal, write the code, select events twice over 81 million events, and analyse the results.
I hope PhD students will be able to do this as well, and use the remaining 2 years and 11 months of their scholarship developing and learning high energy physics.

Results from Benchmark 2 using discriminate function: [doc] [pdf] [ps]

Benchmark 3 - Searching for anti-deuteron.

The idea of this grid benchmark is to look for anti-deuterons in all tracks from all events. This job requires a huge amount of computing and storage resources, and will provide all necessary information to study grid performance and bottlenecks.

There are evidences of anti deuterons production in HERA and argus (1,2). There are also recent models about coalescent process.

  1. Can heavy stable charged particles be produced at BaBar?

Benchmark 4 - Evolutionary neutral pion discriminate function.

This benchmark applies functional gridification in genetic programming evolving discriminate functions to distinguish neutral pions from background in hadronic tau decays using EasyGrid job submission system. Accurate discrimination is important to obtain high purity samples with small systematic errors that allows ratify theoretical predictions with experimental results. Four different data selections criteria according to gammas energies and conventional cut were used to validate our method. A comparison between real data using the same data selection criteria and the discriminate shows our approach is better to discriminate real neutral pions from background than the conventional approach. Monte Carlo and experiment data adherence on neutral pion energy distribution proofs methods effectiveness. The complete analysis is available at Elementary particle identification using evolvable discriminate function and grid and IoP2007 poster has the beatiful plots.

EasyGrid performance and Grid bottlenecks.

Pi0 project, eta decays with generic root application, and searching for anti-deuteron benchmarks provided enough data to support EasyGrid as a submission system for high energy physics in general.

EasyGrid provides the output (if the job runs OK), or abend listings (if the job fails), or abort listings (if grid fails) in 100% of time. Users will have all necessary information in dataset.histo file to find out what to do, and ask for support.

Programming techniques provided in the technical report allows use grid efficiently.

Despite I have succeeded doing distributed analysis, I still have the following concerns about grid implementation by GridPP project:


Project start point: BabarGrid
     Roger Barlow's talk at GridPP10 (Switzerland, 2nd June 2004).

General Overview: Enabling Grid Computer for HEP [ppt] [ps]
     Talks at BabarGrid UK (Manchester, 8th Sept 2004), 
*              GridPP11 (Liverpool,14th Sept 2004), and
              BaBar Meeting (Dresden, 20th Sept 2004)

New BaBar Software Installation (This project was cancelled by Roger Barlow)
    Group meeting 27/10/2004 at Manchester [ppt] [pdf]

Christmas meeting / Manchester 2005: [ppt] [ps]

MetaData Workshop / Grenoble France 2005: [ppt] [pdf] 

Grid2005 Workshop / Seattle, Washington, USA 2005 (preliminar): [doc]  

ICFA Workshop / Daegu, Korea 2005: [ppt]  

Babar Collaboration Meeting / Elba, Italy 2005: [doc]
  I will use the web pages and run grid remotely alive to show the resources and how easy is to use it.  

Babar Grid UK Meeting / Manchester, 2nd June 2005: Tier1 Grid from users point of view: urge of standards

Internal report: How to succeed using grid in High Energy Physics experiments.

BaBar Collaboration,Grid Applications for High Energy Physics Experiments 
Accepted to the Grid 2005 - 6th IEEE/ACM International Workshop on Grid Computing, November 13-14, 2005, Seattle, Washington, USA

Babar Grid UK Meeting / Liverpool, 30th November 2005: EasyGrid, LCG testbed and gridification techniques [Ppt] [Ps] [PDF]

Christmas meeting / Manchester 2005: [ppt] [pdf]

Gridpp15 meeting - Rutherford Appleton Laboratory: ppt

IoP 2006 - University of Warwick: ppt pdf ps 
Also available at GridPP web page (third poster from third row)

GridPP17 - National e-Science Centre - Edinburgh Distributed analysis  

Christmas meeting Manchester 2006: 
    Neutral Pions Discriminate Function

Gridpp18 meeting - University of Glasgow: 
     BaBar Grid Project

IoP 2007 - University of Surrey: ppt 

EGEE User Forum Manchester 9-11 May, 2007 PPT PDF

Registered at EGEE-II Technical Pages - Related projects

GridPP Report: 
    3Q2004 [PDF] [Doc]
    4Q2004 [PDF] [Doc]
    1Q2005 [PDF] [Doc]
    2Q2005 [PDF] [Doc]
    3Q2005 [PDF] [Doc]
    4Q2005 [PDF] [Doc]
    1Q2006 [PDF] [Doc]
    2Q2006 [PDF] [Doc]
    3Q2006 [PDF] [Doc]
    4Q2006 [PDF] [Doc]
    1Q2007 [PDF] [Doc]

GridPP Logbook

Complementary References


To all my coleagues that helped me in this achievement, and Professor Roger Barlow/GridPP for funding the project (PPARC reference PP/C502006/1).

Last modified:
Copyright 2004 Manchester University
Feedback to: jamwer@hep.man.ac.uk