Beam Test Pipeline and Processing Chain

Generally stated, a pipeline consists of tasks and processes created and chained together to implement a multitasking environment on a server. Major components normally include such things as:

• Scheduler/Rules engine
• Task file (e.g., XML) import/export
• Task Database
• Data Catalog database
• Processing history database
• Web interface
• Batch submission interface
• Conditions interface (disk space/usage monitoring, etc.)

Terms used in Pipeline processing:

• Pipeline Application - A program created to define a set of task processes (TPs) that are submitted to the pipeline scheduler in the form of a task file.

• Job - A single unit of batch submission, usually a script submitted to the batch system. Current pipeline processing consists of one batch submission step.

• Run - A number used to label a collection of event data, usually contained in a single file (or multiple files if the amount of data exceeds the maximum file size) for a given data type, e.g., Merit tuples, MC, Digi and Recon ROOT trees.

• Task - Data generated from the same code and configuration. Typically, a task contains a large number of runs. Task names are stored in the Pipeline Database and can be used to generate a list of corresponding data files.

• Release (e.g., BeamtestRelease) - A complete build of the BeamtestRelease checkout package. A primary build target is the gleam.exe program.

GLAST Pipeline

The current GLAST Pipeline (a.k.a. Pipeline I) is a system for scheduling and tracking the execution of data processing jobs on SLAC's CPU farm.

Note: Pipeline II is currently under development with enhancements that, among other things, will support more flexible task scheduling, parallel execution of tasks and subtasks, etc.

In the abbreviated lists shown below, Pipeline I includes an Oracle database, Web Applications, and Scripts.

Also listed above are examples of:

• Pipeline Applications - each one, a particular configuration of data processing jobs submitted to the pipeline scheduler in order to obtain desired output(s).

• Data Utilities that can be invoked to manipulate data before it is processed.

• Users Analysis Tools for analyzing datasets produced by the pipeline.

This section will focus only on the relationship between Pipeline I and two pipeline applications (Monte Carlo and LAT Data) in support of the Beam Test at CERN.

Examples: Pipeline Applications

Monte Carlo Pipeline (for Beam Test)

Steps in defining the Monte Carlo pipeline application included:

1. Identifying disk space.
2. Setting up pipeline config(uration) files.
   1. Defining tasks in XML; uploading the .xml task file(s) to Oracle DB via a web page.
   2. Creating the main perl script.
   3. Creating a jobOptions file.
3. Creating run(s) by:
   • causing entry(ies) in Oracle DB, or
   • causing batch job submission(s)

Beam Test Pipeline (CERN)

The Beam Test Pipeline Application consists of a configuration of data processing jobs submitted to the Pipeline I scheduler. While the process of defining a LAT Data pipeline application follows the same basic steps as those followed in defining an MC pipeline application, a LAT Data pipeline tends to be much more complex due to the number of interrelated task processes (TPs) and their respective dependencies. (See Beam Test Pipeline Application flow chart.)

Once the first task (updateELogDB) is started and exported from Pisa or CERN, it's all automatic unless something fails.

All online data produced by LATTE (currently stored in directories associated with runs numbers) are automatically retrieved and FASTCopied to the SLAC farm. After that, an ORACLE database is populated which provide queries to the data. The pipeline also creates reports, and launches data processing/reconstruction code to produce data files and high level analysis ntuples.

The normal sequence can be summarized as follows:

1. From LATTE, format to digi.
2. Generate digiReport.
3. Run recon on the digi file.
4. Generate reconReport.
5. Generate BT tuple.
6. At all stages populate the ElogDatabase when appropriate.

Notes:

• The sequence of steps within a beam pipeline task is set by an XML file uploaded via the GINO front end to create the task. This file also sets up input and output files, what code to run, which batch queues to run it on, etc. The XML files are generated by perl scripts in the task directories under $beatestPlRoot.

• Tasks launch other tasks using the $beamtestPlRoot/lib/TaskLaunch.pl script. By convention, each task is launched in a separate task process (TP). The TP, script, and the wrapper it uses have names beginning with "Launch".

• There is no file, script, datastructure, or anything that codifies the global structure; tasks launch other tasks without GINO understanding it.

Beam Test Processing Chain

The Beam Test Pipeline Application is a part of the Beam Test Processing Chain shown below and consists of a configuration of data processing jobs submitted to the Pipeline I scheduler.

Note that the Beam Test Pipeline submits jobs to the Pipeline I scheduler which then schedules the jobs on the SLAC CPU farm. Once the first task (updateELogDB) is started and exported from Pisa or CERN, it's all automatic unless something fails.

Beam Test Data Processing Chain

Reading Ancillary Data

Ancillary data fields are read from rcReport, but they are not put in a file. However, they can be read by a database query (e.g., ./lib/queryElogReportTable.pl 700000597 beam_momentum), but only on SLAC machines.

Note: Runs prior to 700000529 did not have any of the new fields entered.

Beam Test Data Monitoring

To see if a run is processed:

1. Go to: Beam Test Data Monitoring.

2. Enter the version of the current set of beamtest tasks (e.g. v1r030603p4) and click on the Filter button. If none of them are "Waiting", "Running", or "Finalizing", they are done.

Also see: GLAST Pipeline Front End (PFE) tutorial.

Test a New Version of a Script

To test a new version of a script:

1. Instead of installing the scripts on ~glast/pdb_config/dpf_config_prod.csh, install the new script on the test pipeline server:

   source ~glast/pdb_config/dpf_config_test.csh

2. Reprocess an old run by running:

   $beamtestPlRoot/online/BeamTestLaunch.pl $runId

Tip: Everything should be set up in: $beamtestPlRoot/setup/svacPlSetup.cshrc

Note: Instructions for installing a new version of the scripts can be found in:

$beamtestPlRoot/doc/install.txt

Troubleshooting

Tracking Down a Problem

1. First go to the Pipeline Front End (PFE) and determine where it failed:

   http://glast-ground.slac.stanford.edu/Pipeline/

2. Then look at the log file to determine which task process that failed.
   (Refer to the GLAST Pipeline Front End (PFE) tutorial.)

Tip: A failure is often due to a transient problem and can be fixed by:

$PDB_HOME/rollBackFailedRun.pl $task $run

Note: If the failure is due to a bug in the pipeline code, and you choose to check the code out and modify it:

1. First, test the bugfix on the test pipeline server.

2. Then ask the Change Control Board (CCB) for permission before installing the bugfix in production code.

Halting Pipeline Processing

It is safe to turn off injection by touching the "halted" file while runs are being processed. Runs that were already injected will continue to be processed, but new runs will not be injected until the file is removed.

Caution! Do not change the "current" link while runs are being processed.

Reprocessing a Run

To reprocess a run:

1. Delete the old links :

   $beamtestP1Root/lib/deleteLinks.csh $run

2. Then enter:

   $beamtestP1ROOT/online/BeamTestLaunch.p1 $run

Note: If reprocessing runs stored on an older disk, create a script to reprocess runs stored on the old disk by copying BeamTestLaunch.pl, then create links to the old data before launching createRun.pl. BeamTestLaunch.pl only works for runs where the LDF file is on the current raw data disk.

For example, at the time this document was written, the current disk was u37; however, the ldf files taken to date were written to u30, and a script ($beamtestPlRoot/online/u30Launch.pl) was created to reprocess those runs.

Tips:

• Code can be checked out from:

/afs/slac.stanford.edu/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline/versionNumber
(e.g., v1r030603p4)

• To view install.txt, go to the CVS Repository, Index of /BeamTestPipeline/doc

Notes:

• Do not set the environmental variable for PIPELINE_HOME; that variable is used only within the installation instructions.

• However, do set the following variable:

setenv beamtestPlRoot /afs/slac.stanford.edu/g/glast/ground/PipelineConfig
/BeamTest-tasks/beamtestPipeline/versionNumber
(e.g., v1r030603p4)

(The line shown above is wrapped.)

.../current/doc/install.txt

Installing a new version of the SVAC pipeline
---------------------------------------------

<PIPELINE_HOME> currently ==
/afs/slac.stanford.edu/g/glast/ground/PipelineConfig/BeamTest-tasks

1. Currently, Beam Test Pipeline code resides at:

/afs/slac.stanford.edu/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline

There you will find two symbolic links labeled:

• Current - which points to the currently installed version of the code

• test - which points to the test pipeline version

??????????????????????

Turn off the pipeline (
cd <PIPELINE_HOME>
echo Manual disable by `whoami` at `date` > halted
chmod g+w halted
).

??????????????????????

Check out the code (BeamTestPipeline in /nfs/slac/g/glast/ground/cvs).

Copy BeamTestPipeline directory to
<PIPELINE_HOME>/beamtestPipeline/<tag>.

Make your edits.

1. From CVS, checkout the last tagged version into a new directory with an incremented version number.

Make your edits in the new version.

Make sure that svacVersion in setup/svacPlSetup.cshrc is the same as the tag
you will be using <tag>.

Make sure that the line containing "setenv eLogTestOnly 1" in
svacPlSetup.cshrc is commented out.

If you are changing BT release, you need to modify:

• EngineeringModelVersion (actually the BeamTestRelease version) and
• sasLocation, (both near the top of svacPlSetup.cshrc) and probably
• Em2Version (in the digitization section; this is misnamed, it's actually the version of Gleam).
• Also check joVersion ( the version of beamtest06) and
• RunRootAnalyzerVersion (the version of BeamTestTuple).

3. Review the instructions in .../current/doc/install.txt.

In particular, note that in the file .../setup/svacPlSetup.cshrc you will have to change the versions of the:

• Pipeline
• beamtest06
• BeamTestTuple
• BTRelease

plus any other changes that may be specified in the install.txt file.

3. Make sure the line containing "setenv eLogTestOnly 1" in svacPlSetup.cshrc is commented out.

?????????????????????????????

If you are changing disks, you need to go to
/afs/slac/www/exp/glast/ground/LATSoft/nfsLinks and make a symbolic link to
the new disk.

?????????????????????????????

4. Commit your changes.
5. Tag it <tag>.
6. Set the beamtestPlRoot environment variable to point to the new version you have just tagged. For example:

setenv beamtestPlRoot afs/slac.stanford.edu/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline/v1r030603p7

Note: The line above is wrapped.

7. Perform: source setup/svacPlRoot

Run $beamtestPlRoot/*/genXml.pl. This will create several XML files in the
current directory ($beamtestPlRoot is a good choice) with names like
task-version.xml.

8. Generate xml files ???for the definition of the pipeline??? by running the perl script */genXml.pl in each of the following subdirectories:
   • configReport
   • digitization
   • digiReport
   • recon
   • eLogUpdate
   • reconReport
   • svacTuple
   • tkrReport

Upload the XML files created in the previous step with web front end at
http://glast-ground.slac.stanford.edu/

9. Upload the 8 files created above to the test pipeline first at:

   http://glast-ground.slac.stanford.edu/Pipeline/

???Don't forget to select Prod or Test as you decided to upload... on the right corner before uploading configuration files.??????????

10. Redefine the symbolic link for "test" to the one you are updating.
11. Test on an existing run:

$beamtestPlRoot/online/BeamTestLaunch.pl 700001121

Note: The pipeline will generated the corresponding files, but without updating the links on the database.

12. Look at the test pipeline and verify that everything is correct.
13. If you decide to update the pipeline and haven't already done so, check with the Change Control Board. and email the beamlist.
14. Before updating the prod, create the "halted sentinel file that stops the injection of data into the pipeline at: /afs/slac.stanford.edu/g/glast/ground/PipelineConfig/BeamTest-tasks using the command:
15. echo Manual disable by 'whoami' at 'date' > halted chmod g+w halted.
16. Update the xml files on prod pipeline.
17. Be sure you have commented out the disabling of elog updates in teh svacSetup.cshrc ile and everything is okay.
18. After verifying that all processing for previous runs has completed, change the "current" symbolic link to point to the new version.
19. Remove the halted file.
20. If you decide to reprocess an existing run, first remove the old links by entering:

    $beamtestPlRoot/lib/deleteLinks.csh $run

then enter:

$beamtestPlRoot/online/BeamTestLaunch.pl $run

 

 

Wait for the pipeline to empty. Use the web front end to see that all runs
for all current tasks are done. This could take hours.

Replace the symbolic link <PIPELINE_HOME>/beamtestPipeline/current with one that
points at <tag>. Or if you're using the test server, change the
<PIPELINE_HOME>/beamtestPipeline/test link instead.

Turn the pipeline back on
(rm <PIPELINE_HOME>/halted).

Upgrading to a New Version of the BT Pipeline

Caution! When it is necessary to upgrade to a new version of the Beam Test Pipeline, first get permission from the Change Control Board (CCB). Be sure to test your install of the new version before cutting over, and never cutover while runs are still being processed.

1. To check for the current and test versions of the Beam Test Pipeline code, go to:

/afs/slac.stanford.edu/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline

There you will find two symbolic links labeled:

• Current - which points to the currently installed version of the code

• test - which points to the test pipeline version

2. From CVS, checkout the last tagged version into a new directory with an incremented version number.

Make your edits in the new version.

3. Review the instructions in .../current/doc/install.txt.

In particular, note that in the file .../setup/svacPlSetup.cshrc you will have to change the versions of the:

• Pipeline
• beamtest06
• BeamTestTuple
• BTRelease

plus any other changes that may be specified in the install.txt file.

.../current/doc/install.txt

Installing a new version of the SVAC pipeline
---------------------------------------------

<PIPELINE_HOME> currently ==
/afs/slac.stanford.edu/g/glast/ground/PipelineConfig/BeamTest-tasks

1. To check for the current and test versions of the Beam Test Pipeline code, go to:

/afs/slac.stanford.edu/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline

There you will find two symbolic links labeled:

• Current - which points to the currently installed version of the code

• test - which points to the test pipeline version

??????????????????????

Turn off the pipeline (
cd <PIPELINE_HOME>
echo Manual disable by `whoami` at `date` > halted
chmod g+w halted
).

??????????????????????

Check out the code (BeamTestPipeline in /nfs/slac/g/glast/ground/cvs).

Copy BeamTestPipeline directory to
<PIPELINE_HOME>/beamtestPipeline/<tag>.

Make your edits.

2. From CVS, checkout the last tagged version into a new directory with an incremented version number.

Make your edits in the new version.

Make sure that svacVersion in setup/svacPlSetup.cshrc is the same as the tag
you will be using <tag>.

Make sure that the line containing "setenv eLogTestOnly 1" in
svacPlSetup.cshrc is commented out.

If you are changing BT release, you need to modify:

• EngineeringModelVersion (actually the BeamTestRelease version) and
• sasLocation, (both near the top of svacPlSetup.cshrc) and probably
• Em2Version (in the digitization section; this is misnamed, it's actually the version of Gleam).
• Also check joVersion ( the version of beamtest06) and
• RunRootAnalyzerVersion (the version of BeamTestTuple).

3. Review the instructions in .../current/doc/install.txt.

In particular, note that in the file .../setup/svacPlSetup.cshrc you will have to change the versions of the:

• Pipeline
• beamtest06
• BeamTestTuple
• BTRelease

plus any other changes that may be specified in the install.txt file.

3. Make sure the line containing "setenv eLogTestOnly 1" in svacPlSetup.cshrc is commented out.

?????????????????????????????

If you are changing disks, you need to go to
/afs/slac/www/exp/glast/ground/LATSoft/nfsLinks and make a symbolic link to
the new disk.

?????????????????????????????

4. Commit your changes.
5. Tag it <tag>.
6. Set the beamtestPlRoot environment variable to point to the new version you have just tagged. For example:

setenv beamtestPlRoot afs/slac.stanford.edu/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline/v1r030603p7

Note: The line above is wrapped.

7. Perform: source setup/svacPlRoot

Run $beamtestPlRoot/*/genXml.pl. This will create several XML files in the
current directory ($beamtestPlRoot is a good choice) with names like
task-version.xml.

8. Generate xml files ???for the definition of the pipeline??? by running the perl script */genXml.pl in each of the following subdirectories:
   • configReport
   • digitization
   • digiReport
   • recon
   • eLogUpdate
   • reconReport
   • svacTuple
   • tkrReport

Upload the XML files created in the previous step with web front end at
http://glast-ground.slac.stanford.edu/

9. Upload the 8 files created above to the test pipeline first at:

   http://glast-ground.slac.stanford.edu/Pipeline/

???Don't forget to select Prod or Test as you decided to upload... on the right corner before uploading configuration files.??????????

10. Redefine the symbolic link for "test" to the one you are updating.
11. Test on an existing run:

$beamtestPlRoot/online/BeamTestLaunch.pl 700001121

Note: The pipeline will generated the corresponding files, but without updating the links on the database.

12. Look at the test pipeline and verify that everything is correct.
13. If you decide to update the pipeline and haven't already done so, check with the Change Control Board. and email the beamlist.
14. Before updating the prod, create the "halted sentinel file that stops the injection of data into the pipeline at: /afs/slac.stanford.edu/g/glast/ground/PipelineConfig/BeamTest-tasks using the command:
15. echo Manual disable by 'whoami' at 'date' > halted chmod g+w halted.
16. Update the xml files on prod pipeline.
17. Be sure you have commented out the disabling of elog updates in teh svacSetup.cshrc ile and everything is okay.
18. After verifying that all processing for previous runs has completed, change the "current" symbolic link to point to the new version.
19. Remove the halted file.
20. If you have to reprocess an existing run, first remove the old links by entering:

$beamtestPlRoot/lib/deleteLinks.sch $run

Then enter:

$beamtestP1Root/online/BeamTestLaunch.pl $run

 

 

Wait for the pipeline to empty. Use the web front end to see that all runs
for all current tasks are done. This could take hours.

Replace the symbolic link <PIPELINE_HOME>/beamtestPipeline/current with one that
points at <tag>. Or if you're using the test server, change the
<PIPELINE_HOME>/beamtestPipeline/test link instead.

Turn the pipeline back on
(rm <PIPELINE_HOME>/halted).