Download Manager

The Download Manager allows files to be securely downloaded from the data catalog. It has two variants:

• GUI based version – Invoked directly from:
  • the Data Catalog web interface;
  • Data Catalog's Key Locations, and from the
  • Data Processing Monitor.

• Line Mode Version (Tip: You can use this line mode tool to download files to remote sites using a cron job.)

Note: The download manager checks if the files to be downloaded already exist. If so they will not be downloaded again. If a newer version exists it will replace the existing version (and the existing version will be renamed *-old-n).

Data Catalog GUI-based Version

The Data Catalog is a database designed for tracking LAT datasets, and can be used with:

• Disk files in AFS, NFS, or XROOTD servers, or tape archives.
• Data created inside or outside of the processing pipeline.
• Data created/stored at SLAC, or elsewhere.
• One or more locations per dataset.

The catalog simplifies access to data by providing a uniform view of files, regardless of their physical location. Data can be organized into a tree of "virtual" folders, which do not have to correspond with the physical location of data. Data can also have associated "meta-data", some of which is required and verified by the catalog software (e.g., size, location, run range, and creation date). Other meta-data is user-defined and is arbitrarily extensible.

Data can be browsed using virtual folders and by "groups", which contain a homogeneous list of datasets, and it can be searched using meta-data.

Note: As new datasets are registered, the Data Crawler validates the files, and extracts meta-data such as file size, number of events, etc.

	The following GUI will be displayed:
Click OK to launch the Download Manager.

The Download Manager GUI will be displayed:

Launch Download Manager from Key Locations

Tip: Note the Key location shortcuts in the Name column when you first go to the Data Catalog:

Launch Download Manager from the Data Processing Monitor

Tip: The Download Manager can also be launched from the Data Processing monitor.

Line Mode Version

Note: You can use this line mode tool to download files to remote sites using a cron job.

Prerequisite: Java version 1.5, or greater.

1. Generate a key file.

Generate a key file to authenticate yourself to the server. To do this simply click on this link:

Generate Key File

Notes:

• If you have not already done so, you will need to login with your SLAC username/password . Save the downloaded key file as:

~/.downloadManager/glastKeyFile

Remember to set the permissions on the file/directory so it can only be read by the user who will run the command line download manager.

2. Download the launcher jar file, and store it wherever you like.

3. Run the download manager from the directory where you saved launcher.jar by issuing the following command:

java -jar launcher.jar --folder=/Data/Flight/Level1/LPA --group=FT1 --destination data/FT1

Arguments: The full list of arguments supported is:

 -destination,--d <arg>   Destination folder   
 -folder,--f <arg>        Folder to download   
 -group,--g <arg>         Group to download   
 -help,--h                Display usage information   
 -keyFile,--k <arg>       File containing download key   
 -nDownloads,--n <arg>    Number of simultaneous downloads   
 -q,--quiet               Turn off informational messages   
 -runMax <arg>            Maximum run number   
 -runMin <arg>            Minimum run number   
 -x,--debug               Turn on debug output