SCons Release Manager

Just like the CMT Release Manager, the SCons Release Manager is split into several components. These components are the batch submission system, the workflow system, and the release manager system. Unlike the CMT Release Manager, the SCons Release Manager is not written in perl. It is written in C++ relying mostly on the Qt library. Additionally, it depends on the Boost Spirit library, and the stx-exparser library. The Qt library is used for most platform independent interactions. The Boost spirit and stx-exparser are used on the linux side only for linux components only.

Qt

A major component of the SCons infrastructure is the Qt library. For linux and the mac the source code is obtained from the trolltech website and compiled. In both cases it's compiled as a static library. For windows the installation is a bit more complicated. The windows installation doesn't come with the mysql plugins pre-compiled. As a result it needs to be compiled after installing Qt for windows. From the trolltech webpage one needs to download the LGPL/Free version for windows. There are several choices to download. The appropriate one is the Framework Only version for Windows. This version is complied with mingw so the machine it's installed on will need mingw. The installer for Qt will take care of installing it but it should NOT be installed on any of the machines that are used by the ReleaseManager; SCons might get confused with mingw and visual c++ both installed.

Once the Qt installer is done, the mysql plugin needs to be compiled for Qt. For instructions on how to compile Qt 4 MySQL drivers with MinGW, see Building QT4 MySQL Drivers with MinGW.

Batch Submission System

Just like its CMT counterpart, the Batch Submission System is used to submit jobs to lsf, monitor the jobs, and notify the layers above of finished jobs. The information for the Batch Submission system is stored in a MySQL database on mysql-node03.slac.stanford.edu. Additionally, Batch Submission System consists of a c++ library and two applications which obtain their information from the database.

Just like the CMT counterpart, the Batch Submission System accepts jobs to execute from the upper layers of the system, executes them on lsf, and notifies the upper layer of state changes.

Database

The MySQL database for the Batch Submission System can be accessed by the rd_lsf_u user and the glastreader user. The database name is rd_lsf. The database contains the following tables:

arguments	Contains the arguments to be passed to the program when executing on lsf.
callback	Contains the details on which state changes the upper layer should be notified.
callbackArgs	Contains arguments to pass to the callback program when executing it.
command	Contains the command to execute on lsf.
job	Contains contains meta data, such as when it was registered, on commands to be executed on lsf.
lsfOptions	Contains any options to be passed to lsf when submitting.
output	Contains the output of the lsf job.
run	Contains the details of a job's attempt at executing on lsf.
settings	Contains global settings for the Batch Submission System.

Applications

The Batch Submission System consists of two applications. The lsf Daemon and the lsf callback application. The daemon is setup to run on glastlnx20 via trscron. It checks the database for new jobs. Jobs are submitted to lsf in suspend mode. For each job, there are three lsf tasks submitted. The first tasks is the actual command to be executed. The second task is a call to lsf callback application with the condition that it be executed as soon as the first task starts execution. The third task is a call to lsf callback application with the condition that it be executed as soon as the first task finishes execution. All lsf tasks are submitted and initially put into suspend mode. If all three tasks are submitted successfully, the jobs are changed from suspend to pending mode so they actually start execution. If at any time anything goes wrong, the lsf tasks are killed, and the run in the table is marked as failed. The job is put back into the queue to be attempted at a later time as a new run.

The lsf callback application is called once when the lsf task starts and once when it finishes. When started, it updates the run table with the information about the start time. Additionally, it checks the callback table to determine if the higher layers wish to be notified when the job has started. If so, it calls the callback command to notify the higher levels. When called for a task that has finished, the callback system checks the lsf output, parses some of the information, like the return code, and stores that information in the run, job, and output tables. Additionally, it checks for any callback information for the higher layers.

Settings

The batch submission settings are controlled by a settings table with a bunch of name/value pairs. The valid names that can be specified is currently not documented anywhere except in code. The code can be viewed in CVS; see: View of /grits-cpp/src/lsf/lsfDaemon/JobSubmitter.cxx

The function JobSubmitter::callBsub() is the one that parses these name value pairs and creates the correct bsub command line options.

Workflow

The Workflow system is a rule based script execution system, with each script considered a stage in the workflow. The workflow moves from one stage to the next by evaluating rules set forth for each stage, and the rules and stages for the workflow are stored in a database on mysql-node03. The workflow consists of a static library and a single executable. This library is used by other systems that wish to use the workflow for initial submission and for controlling various settings of the workflow runs. The executable is run after each stage of the workflow finishes execution in the batch submission system described above, and it computes which new stages need to be executed based on the rules described for the finished stage.

Workflow Database

The database is stored on mysql-node03 and the database name is rd_workflow. The database contains the following tables:

Tables_in_rd_workflow batchOptsOverride conditions run runArgs runScripts runScriptsOverride runSettings settings workflowScripts workflows

Each table performs a unique function as follows

Library

The static library for the workflow system contains a set of functions that allows C++ programs to create new workflow runs and for specifying settings for these runs. The source code and the available functions can be viewed in CVS and is also compiled and available at:

~glastrm/grits-cpp/src

The library makes use of the Qt libraries and uses qmake as its build system. The Qt library currently used is version 4, and it's installed in ~glastrm/externals, which is a set of symlinks using AFS @sys variable to the appropriate platform installation in nfs.

Executable

The workflow system has a single executable called workflowCallback, which is executed by the batch submission system when each stage starts executing and stops executing. When called for the start of the execution of a stage, it simply updates the mysql tables to indicate the start time. When executed to indicate the stop of execution of a stage, it checks the condition table to determine which, if any, other stages need to be executed. If new stages for execution are found, they are submitted to the batch submission system, and the cycle is repeated until a particular run has no more stages to execute.

Release Manager

Just like in the CMT version, the Release Manager consists of a few scripts that are registered in the workflow system as various stages. Conditions are setup in the workflow stages for which stages to execute next. Additionally, the Release Manager consists of several mysql tables.

Release Manager Database

The Release Manager database is stored on mysql-node03. The database name is rd_releasemgr. The database contains the following tables:

Tables_in_rd_releasemgr build buildPackage extLib os outputMessage package settings subPackage subPackageCompileFailures variant versionType

The explanation of each of the tables is as follows

build	Contains the platform dependent information for a particular build.
buildPackage	Contains the platform independent information for a particular build.
extLib	Contains the external libraries used for a particular build.
os	Contains a list of operating systems currently supported.
outputMessage	Contains the checkout or compile output for a particular build.
package	Contains a list of checkout packages currently supported.
settings	Contains both global and build specific settings for the Release Manager.
subPackage	Contains a list of subPackages and their compile status for a particular build.
subPackageCompileFailures	Contains the location of specific compile failures in the compile output for a particular subPackage.
variant	Contains a list of supported variants.
versionType	Contains a list of supported versionTypes.

The os, variant, versionType, and package tables define a list of possible combinations of what the Release Manager can compile or keep track of. When new packages, oses, etc. are created, they need to be inserted in these tables. Each of these tables will create automatic unique IDs that are referenced by the other tables. The settings table references both platform independent settings and platform dependent settings. Platform independent settings have NULL values for the columns that reference the os or variant tables and only include values for the package and versionType tables. Platform dependent settings have values that refer to os, variant, versionType, and package tables.

Executables

The executables and a short description of what they perform is as follows:

checkoutBuild	This executable checks out code from cvs in preparation to building it.
cleanBuild	Erases parts of a build that are not needed for execution (source code and temporary files).
compileBuild	Builds the code checked out with checkoutBuild.
createDoxygen	Generates doxygen output for a build.
deleteBuild	An interactive program to trigger the RM to erase a build.
eraseBuild	Erases everything belonging to a build and marks the build as hidden in the database.
finishBuild	Marks the build as having finished.
releaseManagerDaemon	Runs in the background submitting new builds to the workflow.
testBuild	Runs the unit tests for a particular build.
triggerBuild	Can be used to trigger a new build.
createReleaseBuild	Creates the tar.gz/zip files of the release and externals for the installer.

The releaseManagerDaemon runs for 24 hours on glastlnx20. In that period it checks for new builds to submit to the workflow based on the rules specified in the settings tables. It is capable of submitting builds for all the combinations that are allowed by the os, variant, versionType, and package tables as long as appropriate settings exist in the settings table. This daemon is started up by trscrontab as the glastrm user on glastlnx20 every 24 hours shortly after the previous instance has quit.

All executables take an argument of --buildId. The buildId references the unique ID stored in the build table.

Notification

The Release Manager notifies users of failures when executing the finishBuild executable. Similar to the CMT RM, the finishBuild executable checks the database for any registered failures and looks up the author's information in the package SConscript file. Unlike the CMT RM, the finishBuild executable attempts to bundle as many failures together for a particular author as it can. The goal is to only send a single email per build to a person for all the packages that belong to the author which contain errors.

Old Style Tagger

The CVS repository is currently set to deny any tags that do not conform to the format packageName-XX-YY-ZZ when the following rules apply:

1. The tagging user is not glastrm.
2. The package being tagged is not in the users directory.

These rules are enforced by the tagger script that's located in:

/nfs/slac/g/glast/ground/cvs/CVSROOT/tagger

The CMT Release Manager still expects old style tags of the form vXrYpZ, so the tags above need to be converted to the old style tags. This conversion is done in two steps. First, the tagger script writes a history of tags to a file located in:

/nfs/slac/g/glast/ground/cvs/CVSROOT/tagHistory

The second part of this is a cron job running every minute on glastlnx14. It executes a perl script located in: ~glastrm/oldStyleTagger.pl

This script reads the tagHistory file and determines what tags have occurred since the last time it ran (1 minute ago). It determines which packages where tagged with the new style tags and converts them to old style tags. It then applies these old style tags to the package. This cron job has to run as the glastrm account; otherwise, the two rules above that deny old style tags will apply, and the tag will be denied. There's also a testOldStyleTagger.pl that is identical to oldStyleTagger.pl except for not actually executing cvs tag commands. It is used for testing out what the oldStyleTagger.pl would do for debugging purposes.