Control Room Computer Update to v10r9p1

  • Updated minerva-cr-01 and minerva-cr-02 to v10r9p1
  • Modified setupFiles for initiating setup with v10r9p1
    • tempSetup.sh
    • .bashrc
  • Modified setup.min.soft.sh for v10r9p1
    • Python was a problem for v10r9p1 setup script
    • Created an alias to use the local version python instead of Framework Version
      • alias python=”/usr/bin/python”
  • Tested and Tagged ControlRoomTools under v10r9p1 as “stable_v10r9p1″
  • Updated the .bashrc for using the setup file under v10r9p1
  • Linked .k5login with “cmtuser/Minerva_v10r9p1/Tools/ControlRoomTools/authenticate/k5login-master”
  • Updated the documentation on wiki: “Control_Room_Setup_Manual”

GMBrowser Problem

  • GMBrowser Live works but shifter can not access to the previous runs and subruns
    • GMBrowser -r xx -s xxx does not work
  • The files are not copied automatically to the /minerva/data/online_processing/swap_area
  • I copied the files manually:
    • Connect to the mnvnearline1 – it has the BlueArc /minerva/ mount
    • Necessary files located: /scratch/nearonline/var/job_dumb
  • This solved the issue for non copied files.
  • I checked the log file for nearline_bluearc_copy.sh script under /scratch/nearonline/var/logs
    • After sometime the auto-script seems to be working.
  • Currently, we are not running any runs, I will check the status again tomorrow

Minerva Software Installed on new CR Machines (ROC-West)

MINERvA Software Installation on minerva-cr-01 and minerva-cr-02

  • Firefox configured for Shifter Bookmarks
  • Special Kerberos Principal Installed
  • ROOT 5.34/21 installed and tested
  • ControlRoomTools Installed and Configured
    • setup.sh file and .k5login file configured with ControlRoomTools
    • GMBrowser installed and tested
  • mnvdaqrunscripts installed and configured
    • registered new hostnames(minerva-cr-01 and minerva-cr-02) into “mnvdaqrunscripts/install.sh”
    • committed and tagged changes as “oaltinok_2014_09_17″
  • mnvruncontrol installed and configured
    •  I tried connecting using run control but could not connected. Will test again tomorrow.
  • MINOS Software is NOT installed

Major Update to RunControl Software (v6r1)

  1. Killed all processes
  2. Jeremy updated the
    • mnvonline0.fnal.gov
    • mnvonline1.fnal.gov
    • mnvonlinelogger.fnal.gov
    • minerva-rc.fnal.gov
  3. I updated the remaining Control Room Computers
    • minerva-evd.fnal.gov
    • minerva-bm.fnal.gov
    • minerva-om-02.fnal.gov
  4. Testing the Updates
    1. Successful Test on Control Room Computers
    2. Successful Test on Rochester UROC
    3. Successful Test on Tufts UROCs
  5. I updated the UROC_sw_manager.py script and notified UROC Users

Fermilab Power Failure

  • On Sunday 03:30 am there was a power failure affecting MINOS and MINERvA underground machines
  • Control Room Computers lost network mount to /minerva/data/
    • GMBrowser needs /minerva/data mounted and it was not working
    • minerva-evd is used by UROCs to mount /minerva/data and they are also affected.
    • Carrie opened a service ticket to ask Computer Division Help for Control Room Computers
    • Computer Division solved the incident and all machines and UROCs working properly.
  • mnvonlinebck1.fnal.gov machine is still down and we have no access to Veto Wall HV Monitoring.
  • e-Checklist can be used either one of the following servers: (minerva-wbm was down due to power glitch)
    • http://minerva-wbm.fnal.gov/minerva/echecklist/mininfo.php
    • http://nusoft.fnal.gov/minerva/echecklist/mininfo.php

minerva-om Update

  • minerva-om no longer support MINERvA Software
  • minerva-om has latest MINOS RMS Installed for om near check
  • .bashrc script modified to prompt users to use the “start_MinosOm.sh” command to start the minos om GUI
  • Nothing removed and every file and software are recoverable.

UROC and Control Room Update

  • Heather and Joel has a new GUI for Veto Wall HV Monitoring
  • I have written a script: RunVetoHVMonitor
    • Script is added to Tools/ControlRoomTools/launchers and Tools/ControlRoomTools/linstall.sh
    • Tagged the version as oaltinok_2014_05_13
    • UROC_sw_manager.py updated to get the latest version of ControlRoomTools
  • Send an e-mail to UROC_LIST to inform users to update their UROCs
  • Updated the Control Room Computers: minerva-bm, -evd,-om-02, -rc
  • There were some local modifications on minerva-rc
    • MakeDST
    • gmbrowser/macros/OccupancyPlots.C
  • Sent and e-mail to inform Control Room People to know the updates

UROC and Control Room Update

  • Phil Rodrigues updated GMBrowser

    • GMBrowser to make it skip empty plots when it’s cycling. This way, it doesn’t spend most of the time showing the shifter empty canvases.
  • Version Tag: rodriges_2014-05-01
  • Updated the UROC_sw_manager.py with the tag
  • Committed to CVS and tagged UROC_sw_manager.py as CURRENT
  • Tested update on Tufts UROC and UROC02
  • Sent and e-mail to UROC-LIST to let people know the update
  • Minerba also updated Control Room Computers

No Power on FNAL Control Room – April 19, 2014

  • UROCs need a mount to minerva-evd for GMBrowser
  • Tools/ControlRoomTools/authenticate/MountRemoteFS script responsible for mounting UROCs to minerva-evd
  • mnvonlinebck1.fnal.gov is the Emergency Server incase there is a problem with minerva-evd
  • New Script: MountRemoteFS_Emergency
    • This script uses the mnvonlinebck1.fnal.gov as the Bluearc Mount point
  • Tested on both UROCs at Tufts and GMBrowser works with a mount to mnvonlinebck1.fnal.gov
  • Updates committed to CVS and UROC Users informed with the update.
  • • Successfully tested Emergency Script: MountRemoteFS_Emergency

Warning Messages

  • WARNING messages disappeared after some time: Slave Nodes need time to be fed with the Modified Nearline.opts
  • Committed new version to CVS, follow this procedure:
  1. Before editing a file always: cvs update -A
  2.  Modify the file
  3. Commit the file: cvs commit -m “I modified the … … ….” DaqRecv/options/Nearline.opts
  4. Tag latest version: cvs tag -F nearline_03_26 Nearline.opts

Warning Messages

Warnings for each run:

scratch/nearonline/var/job_dump/MV_00010209_0031_numib_v09_1403201607.joblog:JobOptionsSvc

WARNING # (216,1): Reassignment of option ‘MessageSvc.OutputLevel’ .

Previously defined at /scratch/nearonline/mirror/mnvsoft/v10r7p3/minerva/MINERVA/

MINERVA_v10r7p3/Top/MinervaConf/options/MinervaApplication.opts: (53, 1).

/scratch/nearonline/var/job_dump/MV_00010209_0031_numib_v09_1403201607.joblog:JobOptionsSvc

WARNING # (281,1): Reassignment of option ‘ElectronicsMapAlg.AuditExecute’ .

Previously defined at /scratch/nearonline/mirror/cmtuser/Minerva_v10r7p3/Tools/DaqRecv/options/Nearline.opts: (266, 1).

  • modified the Tools/DaqRecv/options/Nearline.opts on near line machines in order to stop the WARNING messages.
  • Changes I made:
    • – I have disabled the MessageSvc.OutputLevel = 4, because it is previously assigned by Top/MinervaConf/options/MinervaApplication.opts
    • – ElectronicsMapAlg.AuditExecute assigned two different lines. I have removed the duplicate line, which causes the WARNING message.
  • If the WARNING messages disappear, I will commit the newest version of Nearline.opts to CVS

watchdog.sh Script on minerva-rc

  • watchdog.sh script installed on minerva-rc under $HOME/bin/
  • watchdog.sh script checks any running process in the name of RunControl.py and opens RunControl GUI if it is not running on minerva-rc
  • watchdog.sh script creates a log file $HOME/watchdog.log, which records the time and date when the RunControl runned automatically
  • crontab edited to run this script every minute.

Hints:

  • make script executable using:
    • chmod +x watchdog.sh
  • In order to edit the crontab use:
    • crontab -e
  • crontab environment is not same with the script environment
  • dump local environment to a temp file:
    • env > localenv.output
  • Copy all the lines in “localenv.output” to the crontab (before your command)
  • For running commands which have GUI use
    • export DISPLAY=:0 (or localenv has this value)
    • Carefull with remote connections. Remote connection Displays are different than local displays
  • To run a command every minute write the following line in crontab:
    • * * * * * . $HOME/bin/watchdog.sh

Control Room Update

  • 3 Components updated
  • mnvdaqscripts
    • cvs update –r v3r6
  • mnvruncontrol
    • cvs update –r v5r7p2
  • Tools/ControlRoomTools
    • cvs update –r nearline_cfg_2013_11_13
  • Updates Completed on 10.00 a.m.

Problems & Troubleshooting

  • On minerva-rc run control does not show up after update
    • Jeremy solved the problem
    • Cause of problem is due to someone locally modified the RunControl.py
    • Always check for CVS messages after update
    • Solve all conflicts