Data Catalog Line-mode Client

The line-mode client is a subset of the full Data Catalog Application Programming Interface (API) and can be accessed from the UNIX command line at SLAC and by Jython scriptlet processes in the GLAST Pipeline.

Note: Any Java program running within the SLAC firewall may use this API to take advantage of the full Data Catalog feature set. (See Data Catalog Java API.)

The Data Catalog Line-mode executable is available at:

    /afs/slac.stanford.edu/g/glast/ground/bin/datacat

Help:

• To display the help screen, invoke the executable with no parameters.

• Command-specific help can be obtained by executing:

/afs/slac.stanford.edu/g/glast/ground/bin/datacat -h <command>

Commands Currently Available  

(The usage is similar to CVS)  The following commands are currently available:

• registerDataset (adds a new dataset to the catalog)

datacat registerDataset [-options] <dataset name> <data type> <logical folder> <file path>

Required Parameters:

<data type>	Type of data in the file (merit, MC, DIGI, RECON, etc.)  See Java API child page for a full list.
<logical folder>	Dataset Folder Path under which to create the new dataset.
<file path>	Physical location of file to add to Data Catalog.

Optional Parameters:

Long Form	Short Form	Parameter	Default Value	Description
--name	-n	dataset name	file name	Name to give new dataset in the catalog.
--group	-G	group name	none	Group under which to store the dataset.
--format	-F	file format	file extension	Format of the file. (root, fits, etc.)
--site	-S	site name	SLAC	Site where dataset physically exists (SLAC, SLAC_XROOT, etc.)
--define	-D	"name=value"	none	Define a meta data name/value pair for the new dataset.  This option may be used more than once.  For naming rules, see the Java API child page

Example:

datacat registerDataset -n 000002 -G merit -D nEvt=2500 -S SLAC -F root merit /
ServiceChallenge/Interleave3h-GR-v11r17/runs  /nfs/farm/g/glast/u43/MC-tasks/
Interleave3h-GR-v11r17/data/merit/Interleave3h-GR-v11r17-000002-merit.root

• addLocation - Adds an additional physical location to an existing dataset.  Use this routine to specify that a dataset exists in more than one physical location (i.e., it's on SLAC NFS and in SLAC XROOT.)  Except for <file path> all of the parameters and options are used to identify the existing dataset entry to which you want to add an additional physical location.

datacat addLocation [-options] <dataset name> <logical folder> <file path>

 Required Parameters:

<dataset name>	Name of existing dataset.
<logical folder>	Data Catalog Folder Path under which the dataset lives.
<file path>	Additional physical location of file to add to the dataset entry.

Optional Parameters:

Long Form	Short Form	Parameter	Default Value	Description
--group	-G	group name	none	Dataset Group in the Data Catalog under which the dataset lives.
--site	-S	site name	SLAC	Site at at which the additional physical location exists.

Example: 

datacat addLocation -G merit -S SLAC_XROOT  000002  /ServiceChallenge/
Interleave3h-GR-v11r17/runs  root://glastrdr//glast/mc/ServiceChallenge/
Interleave3h-GR-v11r17/merit/Interleave3h-GR-v11r17-000002-merit.root

• addMetaData - Adds meta data entrie(s) to an existing dataset.
  

datacat addMetaData [-options] <logical folder>

Required Parameters:

<logical folder>

Logical Folder Path where the group or dataset lives, or to tag with meta data if no dataset or group specified.

Optional Parameters:

Long Form	Short Form	Parameter	Default Value	Description
--dataset	-n	dataset name	file name	Name of existing dataset.
--group	-G	group name	none	Dataset Group in the Data Catalog under which the dataset lives.
--define	-D	"name=value"	none	Define a new meta data name/value pair for the dataset.  This option may be used more than once.  (And must be used at least once!)  For naming rules, see the Java API child page

Example:

datacat addMetaData -d 000002 -G merit -D nEvt=2500 /ServiceChallenge/
Interleave3h-GR-v11r17/runs

Xrootd Preload Library

At SLAC the easiest way to read and filter/merge the files is to use the xrootd preload library, which lets the standard FTOOLs be used directly on the files in xroot.

At the time of this writing, the preload library cannot "trick" cfitsio into recognizing root://glast-rdr.slac.stanford.edu/ as anything other than the start of a URL. This may be fixed in future versions of cfitsio (and the FTOOLs) but, in the meantime, that part of the xroot path to the files has to be removed manually.

Assumption: You do not want to download raw files (which in the case described below would be about 1 GB) directly to your computer, AND you only need (CTBCLASSLEVEL==3, zenith_angle<105), which would be about 10 times smaller.

Example: The following command runs fverify on the first file in a reprocessed set:

/afs/slac.stanford.edu/g/glast/applications/xrootd/FITS/bin/xrdprel -g
fverify /glast/test/Data/Flight/Repro/ReproTest7/ft1/r0239557414-000000_ft1.fits

To process all of the files, save the list from the Data Catalog, remove the root://glast-rdr.slac.stanford.edu/ part, and use it like you would any other list of FT1 files.

Notes:

• gtselect will work on this list of files, but ftselect and fselect will not.

• ftmerge will work, meaning that you will have to do the filtering in 2 steps – either:
  • using gtselect to merge the files and make the zenith angle cut and then ftselect to select CTBCLASSLEVEL==3

  OR

  • use ftmerge to make a large merged FT1 file, and then ftselect to apply
    CTBCLASSLEVEL==3 && ZENITH_ANGLE<105.

  The latter might be faster, given that gtselect has to work through tip.

Also see the Xrootd Preload Library page in Confluence.