How to Fix mocTicker
Owned by: Steve Tether
The mocTicker is a packet forwarder originally designed for use in real time during on-orbit contacts, but is now more of a general purpose repeater for telemetry from any source. It's second function is to detect LAT and GBM alerts in incoming telemetry and make log entries directly in Oracle logging tables, bypassing the logCacher+logDbWriter chain. It also writes its own log to stdout. The mocTicker performs some simple checking for incorrect packet sizes, bad timestamps or uncorrectable Reed-Solomon errors.
ISOC servers and workstations run their daemons in the PROD environment, which means that they use:
- The sitedep file sitedep-slac-flight.ini.
- The Spread groups flight.raw, flight.xpl, flight.ctl and flight.log.
- The ISOC_FLIGHT table space on GLASTP.
Note: All machines have the full complement of daemons installed, but many of them should be running on only one machine because they provide flight-ops-wide services. One brings them up on the another machine in case the usual one goes down.
| How to Fix: |  |
Steve Tether's writeup (in Confluence)
Primary Hub (glastlnx11) Fails
To Test: Check status of isoc-servers.
To Fix: Switch the realtime telemetry reception from the primary hub (e.g., glastlnx11) to a backup (glastlnx06). Normally the Mission Operations Center at Goddard and the Burst Alert Processor at Marshall will connect to the MocTicker running on glastlnx11. If glastlnx11 goes down and can't be brought back up the following steps will let glastlnx06 take its place (you will need to be a member of group glast-isoc).
On glastlnx06, bring up the daemons that normally run on glastlnx11:
svc -u /service/{mocticker,injector,exploder,monitor-web,pagemaster,processrt,rates,ttg-web,xplcache}
After you've done this you'll need to tell the MOC and BAP people that they should contact our secondary system. Once glastlnx11 is back up you'll need to reverse the above steps.
