GMBrowser Problem after Power Outage

  • GMBrowser was not working on Control Room Computers
  • The Problem was tbe following:
    • /grid/fermiapp/ disk was not accesible from minerva-cr-01 and minerva-cr-02
  • Opened a Service Desk Ticket and the Fermilab CD mounted the disks
  • After the mount, GMBrowser did not worked with the Local ROOT installation
    • I modified the script to use the ROOT under /grid/fermiapp
    • Built GMBrowser using new ROOT and it started to work


GMBrowser Update – Uses all gates now

  • Previously GMBrowser that shifters look at only uses a fraction of the gates, because the early processing stages (particularly DecodeRawEvent) were slow.
  • Now that we have a faster version of DecodeRawEvent, and we modified GMBrowser to use all gates
  • Modified following parameters in NearlineCurrent.opts in Tools/DaqRecv/options on mnvonlinelogger, to be 100 percent:
    • PdstlPrescaler.PercentPass          = 25;
    • LinjcPrescaler.PercentPass          = 25;
    • NumibPrescaler.PercentPass       = 20;
  • Ran the “” script in all Nearline Machines to get the update
    • mnvnearline1
    • mnvnearline2
    • mnvnearline3
    • mnvnearline4
  • Informed Current Shifter about the update and started GMBrowser at Tufts UROC
    • Will investigate the behavior for some time, until we make this change permanent.

Control Room Computer Update to v10r9p1

  • Updated minerva-cr-01 and minerva-cr-02 to v10r9p1
  • Modified setupFiles for initiating setup with v10r9p1
    • .bashrc
  • Modified for v10r9p1
    • Python was a problem for v10r9p1 setup script
    • Created an alias to use the local version python instead of Framework Version
      • alias python=”/usr/bin/python”
  • Tested and Tagged ControlRoomTools under v10r9p1 as “stable_v10r9p1″
  • Updated the .bashrc for using the setup file under v10r9p1
  • Linked .k5login with “cmtuser/Minerva_v10r9p1/Tools/ControlRoomTools/authenticate/k5login-master”
  • Updated the documentation on wiki: “Control_Room_Setup_Manual”

GMBrowser Problem

  • GMBrowser Live works but shifter can not access to the previous runs and subruns
    • GMBrowser -r xx -s xxx does not work
  • The files are not copied automatically to the /minerva/data/online_processing/swap_area
  • I copied the files manually:
    • Connect to the mnvnearline1 – it has the BlueArc /minerva/ mount
    • Necessary files located: /scratch/nearonline/var/job_dumb
  • This solved the issue for non copied files.
  • I checked the log file for script under /scratch/nearonline/var/logs
    • After sometime the auto-script seems to be working.
  • Currently, we are not running any runs, I will check the status again tomorrow

Minerva Software Installed on new CR Machines (ROC-West)

MINERvA Software Installation on minerva-cr-01 and minerva-cr-02

  • Firefox configured for Shifter Bookmarks
  • Special Kerberos Principal Installed
  • ROOT 5.34/21 installed and tested
  • ControlRoomTools Installed and Configured
    • file and .k5login file configured with ControlRoomTools
    • GMBrowser installed and tested
  • mnvdaqrunscripts installed and configured
    • registered new hostnames(minerva-cr-01 and minerva-cr-02) into “mnvdaqrunscripts/”
    • committed and tagged changes as “oaltinok_2014_09_17″
  • mnvruncontrol installed and configured
    •  I tried connecting using run control but could not connected. Will test again tomorrow.
  • MINOS Software is NOT installed

Major Update to RunControl Software (v6r1)

  1. Killed all processes
  2. Jeremy updated the
  3. I updated the remaining Control Room Computers
  4. Testing the Updates
    1. Successful Test on Control Room Computers
    2. Successful Test on Rochester UROC
    3. Successful Test on Tufts UROCs
  5. I updated the script and notified UROC Users

Fermilab Power Failure

  • On Sunday 03:30 am there was a power failure affecting MINOS and MINERvA underground machines
  • Control Room Computers lost network mount to /minerva/data/
    • GMBrowser needs /minerva/data mounted and it was not working
    • minerva-evd is used by UROCs to mount /minerva/data and they are also affected.
    • Carrie opened a service ticket to ask Computer Division Help for Control Room Computers
    • Computer Division solved the incident and all machines and UROCs working properly.
  • machine is still down and we have no access to Veto Wall HV Monitoring.
  • e-Checklist can be used either one of the following servers: (minerva-wbm was down due to power glitch)

minerva-om Update

  • minerva-om no longer support MINERvA Software
  • minerva-om has latest MINOS RMS Installed for om near check
  • .bashrc script modified to prompt users to use the “” command to start the minos om GUI
  • Nothing removed and every file and software are recoverable.

UROC and Control Room Update

  • Heather and Joel has a new GUI for Veto Wall HV Monitoring
  • I have written a script: RunVetoHVMonitor
    • Script is added to Tools/ControlRoomTools/launchers and Tools/ControlRoomTools/
    • Tagged the version as oaltinok_2014_05_13
    • updated to get the latest version of ControlRoomTools
  • Send an e-mail to UROC_LIST to inform users to update their UROCs
  • Updated the Control Room Computers: minerva-bm, -evd,-om-02, -rc
  • There were some local modifications on minerva-rc
    • MakeDST
    • gmbrowser/macros/OccupancyPlots.C
  • Sent and e-mail to inform Control Room People to know the updates

UROC and Control Room Update

  • Phil Rodrigues updated GMBrowser

    • GMBrowser to make it skip empty plots when it’s cycling. This way, it doesn’t spend most of the time showing the shifter empty canvases.
  • Version Tag: rodriges_2014-05-01
  • Updated the with the tag
  • Committed to CVS and tagged as CURRENT
  • Tested update on Tufts UROC and UROC02
  • Sent and e-mail to UROC-LIST to let people know the update
  • Minerba also updated Control Room Computers

No Power on FNAL Control Room – April 19, 2014

  • UROCs need a mount to minerva-evd for GMBrowser
  • Tools/ControlRoomTools/authenticate/MountRemoteFS script responsible for mounting UROCs to minerva-evd
  • is the Emergency Server incase there is a problem with minerva-evd
  • New Script: MountRemoteFS_Emergency
    • This script uses the as the Bluearc Mount point
  • Tested on both UROCs at Tufts and GMBrowser works with a mount to
  • Updates committed to CVS and UROC Users informed with the update.
  • • Successfully tested Emergency Script: MountRemoteFS_Emergency

Warning Messages

  • WARNING messages disappeared after some time: Slave Nodes need time to be fed with the Modified Nearline.opts
  • Committed new version to CVS, follow this procedure:
  1. Before editing a file always: cvs update -A
  2.  Modify the file
  3. Commit the file: cvs commit -m “I modified the … … ….” DaqRecv/options/Nearline.opts
  4. Tag latest version: cvs tag -F nearline_03_26 Nearline.opts

Warning Messages

Warnings for each run:


WARNING # (216,1): Reassignment of option ‘MessageSvc.OutputLevel’ .

Previously defined at /scratch/nearonline/mirror/mnvsoft/v10r7p3/minerva/MINERVA/

MINERVA_v10r7p3/Top/MinervaConf/options/MinervaApplication.opts: (53, 1).


WARNING # (281,1): Reassignment of option ‘ElectronicsMapAlg.AuditExecute’ .

Previously defined at /scratch/nearonline/mirror/cmtuser/Minerva_v10r7p3/Tools/DaqRecv/options/Nearline.opts: (266, 1).

  • modified the Tools/DaqRecv/options/Nearline.opts on near line machines in order to stop the WARNING messages.
  • Changes I made:
    • – I have disabled the MessageSvc.OutputLevel = 4, because it is previously assigned by Top/MinervaConf/options/MinervaApplication.opts
    • – ElectronicsMapAlg.AuditExecute assigned two different lines. I have removed the duplicate line, which causes the WARNING message.
  • If the WARNING messages disappear, I will commit the newest version of Nearline.opts to CVS