Accessing LPA Data via LAT Data Catalog

In order to access password protected sites, be sure you have followed the "Get Connected" instructions linked from the red navbar.

  1. See:

    On this page, see:

    In Confluence, see:
    Once you've accessed the Data Catalog, click on the Catalog tab, then check the banner and make sure that the following options (banner - upper right) are set in accordance with what you are interested in doing.

In this case, we are looking for LAT Prod(uction) data:

  • Config: OnOrbit
  • Mode: [ Prod | Dev | Test]
  1. Select View: Tree

Note: Observe that the following checkboxes are located at the top of the left column:

If you wish to access MC, beam test, or obsolete data, select the appropriate checkbox(es) and click on the "Update" button. When accessing LAT Physics Acquisition (LPA) data, leave them unchecked.

  1. Open the correct top-level folder; in this case, open the Data, not ASP or FOS) folder.
  1. Navigate to the data you are interested in (e.g., Data --> Flight --> Level1 --> LPA

CAUTION! Be sure you are in the right top-level folder, e.g., Data ... LAT Physics Acquisition (LPA). It is not unusual to find other folders with similar trees. At the time this was written, an IandT --> Level1 folder had a tree very similar to the one shown at right.

See LAT Science Products.

Tip: If you want to browse LPA Data, see: Wired Event Display: View LPA Data.
  1. Scroll down to the folder you are interested in (e.g., FT1).

After clicking on your selection (e.g., FT1), wait for a few moments for the appropriate database to be accessed, and then observe that your selection is displayed in the column to the right.

 

Note: For information on the options shown above, click on:

 

List Files

Notes:

Additional information is provided for each file included in a dataset, including Type, Format, etc.

Files may not be empty, even if the display says they are. In some cases, the dataCatalog may say that a file is empty, when it isn't. This can result when a special script called the "crawler" has not yet detected the information. The crawler operates on its own timescale, trolling for new files in the dataCatalog and trying to interpret their content. In some cases, the crawler may not yet know how to extract meaningful data from a particular file type.

Drill-down links. Name links (e.g., r0249317900) are also included, enabling you to drill down to detailed production/Task information. Additional links are provided as appropriate. See screen captures for Dataset r0249317900 version 0 shown below:

 

Download Files

After selecting the files you wish to download to your machine (e.g., merit), click on the download link. After a few seconds, the Download Manager GUI should appear and, after a slight delay, the download should start automatically.

Note: You must have Java installed on your machine. If not, see Tips on Installing Java.


Tip: This Option GUI only appears automatically the first time you perform a download. To access this GUI in order to change the Download directory for a subsequent download, click on File --> Options in the Download GUI shown below.
Caution! Downloading will not start until you click on the Start downloading button. Downloading is, however, very fast, and the files can be quite large (e.g., 5.7 GB for Group FT1 shown above). Be sure that you have adequate disk space available.

Also see:
 

Dump file list (SLAC)

If you are creating a script file, the option to Dump file list provides an easy way to copy file locations; it also supports mirroring at multiple sites:

 

Dump file list (SLAC_XROOT)

If you are creating a script file, the option to Dump file list provides an easy way to copy file locations; it also supports mirroring at multiple sites:

LAT Data Catalog: Virtual File System

The LAT Data Catalog is a virtual file system maintained in an Oracle database.  Data may be stored at several locations, e.g., SLAC, University of Washington (UW), Lyon (IN2P3), and elsewhere).  The files themselves may be stored:

  • On disk in AFS-, NFS-, or XROOTD-managed servers.
  • In one of several tape archive systems.
  • Or in any combination of the above. 

The Data Catalog simplifies access to data by providing a uniform view of files that is independent of their physical location, and it provides features that are not available in standard file systems, including tagging files with:

  • Meta-data attributes (typed name/value pairs) that provide additional information about the data they contain.
  • Several physical locations, allowing a file to exist in multiple places for more convenient access.

In addition, the Data Catalog maintains a conventional folder structure even as it provides a group structure, which allows files of different pedigree to be separated while coexisting within the same folder.

The Data Catalog also provides access to files by requesting:

  • A file, or a set of files, at a specific location (folder / group).
  • A set of files via a meta data query.

JAVA API. Access to the Data Catalog is provided via a Java API that is is under continued development, and features are added regularly. Any Java program running within the SLAC firewall may use this API to take advantage of the full Data Catalog feature set.  The Java API is available in Confluence.  The API can be accessed by a Line-Mode Client, available from SLAC UNIX machines, and by Jython scriptlet processes in the GLAST Pipeline. For additional information, see the Advanced (yellow navbar) section of the workbook.

Owned by:

Dan Flath

Last updated by: Chuck Patterson 05/10/2010