Access Restrictions and Password Requirements Many maintainance operations require special privileges. Some of these are described below. Logging in to the glast account on glastlnx Many maintainance operations require logging in as glast on one of the glastlnx\ boxes. We should add some ...

AFS Group Structure and Ownership Hierarchy This page is a reference for how GLAST AFS spaces (directories) are protected via "groups" and how those groups are managed. As of this writing (7/18/2008), there are two overlapping schemes for organizing the permissions on GLASTowned AFS directory spaces ...

How it works There's a cron job running on glastlnx01 for each account that can submit batch jobs. The cron job checks the bsub.jobs table on glastDB.slac.stanford.edu for new jobs and uses bsub to submit them. What to check for if things go wrong The contents ...

CAS is the Single SignOn Server used by all GLAST web applicaitons

Confluence is the enterprise wiki used by GLAST and other experiments at SLAC

Confluence and JIRA Upgrade Instructions This is the log from out most recent upgrade of Jira and Confluence # Download:  confluence1.4.1.zip # Download:  atlassianjiraenterprise3.2.1.zi{}p # Unpack both into:  D:\java # Run export on bugs.freehep.org and jira.slac.stanford.edu: d:\fhjira15jun2005.zip ...

log from out most recent upgrade of Jira and Confluence Download confluence1.4.1.zip Download atlassianjiraenterprise3.2.1.zip Unpack both into D:\java Run export on bugs.freehep.org and jira.slac.stanford.edu d:\fhjira15jun2005.zip (902 kB) d:\slacjira15jun2005.zip (2564 kB) Turn off ...

Using crontab on glast unix machines We currently have different crontab files installed under account glast on different glastXX nodes, which makes it easy to accidentally overwrite someone else's crontab entries. To make this a little less ...

Instructions for GLAST CruiseControl # All cruisecontrol information for glast lives in /nfs/farm/g/glast/u10/cruisecontrol linked to by glast/cruisecontrol # A cron job runs to keep cruisecontrol running on glast01 # All checked out projects are in glast/cruisecontrol/checkout Add ...

CruiseControl Instructions Notes:  All cruisecontrol information for glast lives in:  /nfs/farm/g/glast/u10/cruisecontrol linked to by \glast/cruisecontrol A cron job runs to keep cruisecontrol running on glast01. All checked out projects are in:  \glast ...

you are a current or former UNIX or Linux user, then an installation of Cygwin http://www.cygwin.com on your Windows computer may be useful. I have found the Cygwin toolset useful in my work, and include some tips below on how I ...

Failure Symptoms Failure Symptoms \\ HTF: Pipeline HTF: tomcat Servers HTF Tomcat Servers\\ HTF: xrootd \\ HTF: MySQL \\ HTF: Oracle \\ HTF: Confluence HTF Confluence \\ HTF: JIRA Confluence not responding \\       ...

Contacts Primary contact: blee General Information FASTCopy is a commercial filetransfer program from Softlink, LTD. http://www.softlink.com used to exchange data among the elements of the GLAST ground system. Monitoring FASTCopy transfers to and from SLAC can be monitored ...

glast05 server runs Windows Server 2003, and is the main web server for all GLAST web applications

External Software Packages glast06 is a Windows 2003 server running IIS6. It's purpose is to host web service modules for GLAST and to host Release Manager for Windows. Software Package Version Date Installed Installed By Microsoft Visual Studio 2003 (C ...

Primary contact: Matt Langston mailto:langston@slac.stanford.edu Installiong Visual Studio The xweb utility (SCS's webbased appliation for intalling software) does not work correctly on GLAST's servers. They've removed the old "X drive" where application installation media ...

Overview SLAC policy requires Oracle passwords to be changed every six months. If they are not changed the account will become locked out and things will stop working. GLAST currently uses the following Oracle accounts: Account Usage Databases GLASTDPTEST Pipeline, Data ...

Infrastructure "How to Fix" Home This space is the new home for the "how to fix" (HTF) procedures for operations resources (e.g., servers, databases, pipeline, release manager, etc).  It is also the future home for more detailed information on how to set up ...

How to Fix Confluence/JIRA Owned by: Tony Johnson  Confluence is a serverbased, knowledge management tool enabling people to access, create, and edit content for web pages using a browser and the wiki markup language. Test (and fix) in the following order:   ...

How to Fix Data Catalog Owned by: Dan Flath  The Data Catalog is ............... How to Fix: #  Contact Dan Flath. \\ (See Glast Phone List http://glastground.slac.stanford.edu/LatOps/pagesdev/contactlist/isoccontactlist.html.) !htfdataCatalog.png!  Test and Fix   ...

How to Fix the Data Catalog Crawler Owned by: Tony Johnson The data catalog crawler is used to check the integrity of the data catalog by checking each file shortly after it is registered in the data catalog, and by periodically rechecking files to see that they still exist ...

Duplicate run entries for the same delivery ID Sometimes the summary page will show the same run over and over again for the same delivery ID. The root cause may be that the database table ISOCFLIGHT.FCOPYINCOMING has the same downlink ID for two different ...

How to Fix FASTCopy Incoming Owned by: Philip Hart  How to Fix:  \\ 1. Ingest Failure #ingestFailure\\ !htfFastCopyincoming.png! The FCIncoming script writes to the Oracle tables FCOPYINCOMING (one row/tarball), untars the incoming file to: /nfs/farm ...

How to Fix FASTCopy Outgoing Owned by: Philip Hart The FASTCopy Outgoing http://glastground.slac.stanford.edu/FCWebView/?tabMode=Outgoing monitoring page shows the  transfer process states ("new, batchdone, etc.; see diagram below); this is the first piece of information ...

How to Fix Halfpipe Owned by: Philip Hart Note: Halfpipe processing runs entirely in the pipeline and is subject to the standard problems that occur there.  !htfhalfpipe.png! With the following exceptions, there is little that goes wrong with the pipeline: The makeEvt process will, for example, occasionally fail due ...

How to Fix LogWatcher Owned by: Steve Tether

How to Fix mocTicker Owned by:  Steve Tether  The mocTicker is a packet forwarder originally designed for use in real time during onorbit contacts, but is now more of a general purpose repeater for telemetry from any source. It's second function is to detect ...

How to Fix MySql Owned by: Navid Golpayegani MySQL is a multithreaded, multiuser SQL database management system which runs as a server and provides access to multiple databases.  To Test Check: Nagios MySQL Service Group http://glastnagios.slac.stanford.edu/nagios ...

How to Fix Oracle   Brief description of key applications affected by an Oracle failure ...  ??? Tony ??? To Test: Check Nagios Oracle Service Group http://glastnagios.slac.stanford.edu/nagios/cgibin/status.cgi?servicegroup=OracleServers&style=detail. To Fix: If an ...

How to Fix Pipeline II Owned by: Tony Johnson and Dan Flath  PipelineII servers (PROD and DEV) are now running under account glast on glastlnx12, and glastlnx07 respectively, and cron jobs are set up to start the servers after a reboot or crash ...

How to Fix Tomcat Servers Owned by: Tony Johnson Apache Tomcat is a web container that specifes a runtime environment for web components, including services such as: security, concurrency, lifecycle management, transaction, deployment, etc.   It implements Sun Microsystem's ...

How to Fix Xrootd Owned by: Wilko Kroeger  Note:  xrootd is the daemon for the Xrootd data server. The xrootd cluster consists of redirectors and data server. A redirector is the central entrance point for a client. It does not provide any files but it directs a client ...

IIS is the web server used to front all GLAST web applications

JIRA is the issue tracking and project management web application used by GLAST and several other experiments at SLAC

home page for the SAS Ops HowtoFix space. It covers operational items like the pipeline, System Tests, Release Manager and the various windows and linux servers we operate. If you need immediate help, you can contact these people: Emergency ...

Primary contact:Navid Golpayegani mailto:golpa@milkyway.gsfc.nasa.gov

How to Fix LogWatcher Owned by:  Steve Tether  Brief description. How to Fix: # !htfLogWatcher.png

Fermi/GLAST NFS Storage NFS \\mount \\point \\/nfs/slac/g/glast/ server FS size \\TB Status Usage ground surrey01 0.01 cvs repository, etc. NFS \\ mount \\ point \\/nfs/farm/g/glast ...

place to collect obsolete pages before the are deleted

Primary contact: dflath The pipeline scheduler uses crontab, not trscrontab. Pipeline Front End (Web interface) The pipeline web interface runs on the glast tomcat server. See the tomcat page Tomcat for instructions on restarting it. Dropped Tables Sometimes during ...

Author: mattlangston Introduction GINO uses a set of web services for part of its operation. We currently only use the XML uploader functionality right now, but more functionality may be exposed as we move to the second generation of GINO. The purpose ...

Author: mattlangston Introduction We currently use Oracle 9i for the production server, and Oracle 10g for the development and test servers. There are different procedures for recovering data from an Oracle disaster (for example, accientally dropping tables) depending on whether the server ...

All of these tasks you need to be logged into the system and also have the required privileges. If you don't you need to ask Navid to set the appropriate permissions in the database. Deleting a build To delete a build simply click on the Delete link in the top navigation ...

Batch Submission How it works. A cron job is running on glastlnx01 for each account that can submit batch jobs. The cron job checks the bsub.jobs table on{}glastDB.slac.stanford.edu for new jobs and uses bsub to submit them. What to check for if things go ...

SAS Resources by Monitoring Tool (as of 10/13/08): Tomcat Production Servers (Server Monitoring http://glastground.slac.stanford.edu/ServerMonitoring tool) tomcat01 cas Commons GroupManager tomcat02 \\ CalibrationTrending cruisecontrol DC2EventFinder ServerMonitoring tomcat04 GLAST Server Monitoring (tool) tomcat05 elog ...

Servers and Aliases The goal is as far as possible make services easily movable between machines by using DNS aliases for services wherever possible. In some cases (notably fastcopy) fixed IP addresses are required by external constraints. Boldface indicates critical ...

system tests web front end is accessible to the world at large via the glastground IIS server at: http://glastground.slac.stanford.edu/SystemTests/ The backend runs on a tomcat server (currently tomcat 5.5.7 running on web007 web007, although it should move soon to a glast ...

Important locations Tomcat is set up to run on several of the Glast unix servers. The following directories are used: directory use glast/tomcat/scripts Scripts for starting/stopping/configuring tomcat glast/tomcat/BASE55 Tomcat configurations (per host) glast ...

An ISAPI connector is used to bridge between the glastground IIS server and a Tomcat 5 instance running the system tests backend. Configuration Files The current configuration on glast05 is: (note these are currently in {{c:\tony\jakarta}} to keep them distinct from ...

We have a simple backup utility to keep in sync two different nfs partitions. For example, u12 (/nfs/farm/g/glast/u12) is duplicated on u16 (/nfs/farm/g/glast/u16). Currently we run two shell scripts in a cron ...

Most web applications run on the GLAST tomcat servers. The tomcat page Tomcat describes how to restart servers for particular web applications

web007 There are a number of servers running on web007 used by Glast IIS5 This runs with many virtual servers, mostly unrelated to glast. The ones relevant to glast are: jira.slac.stanford.edu confluence.slac.stanford.edu These both simply forward requests to the relevant tomcat server (see below). IIS5 ...

GLAST Windows Servers

xrootd runs on glast01 to serve the root histograms required by the system tests web frontend. All relevant files can be found in glast/xrootd: xrootd The script which runs xrootd xrootd.cf The configuration file for xrootd monitor A program run ...