Major Update to RunControl Software (v6r1)

  1. Killed all processes
  2. Jeremy updated the
    • mnvonline0.fnal.gov
    • mnvonline1.fnal.gov
    • mnvonlinelogger.fnal.gov
    • minerva-rc.fnal.gov
  3. I updated the remaining Control Room Computers
    • minerva-evd.fnal.gov
    • minerva-bm.fnal.gov
    • minerva-om-02.fnal.gov
  4. Testing the Updates
    1. Successful Test on Control Room Computers
    2. Successful Test on Rochester UROC
    3. Successful Test on Tufts UROCs
  5. I updated the UROC_sw_manager.py script and notified UROC Users

Validation

  • University of Minnesota Duluth group updating a pretty old UROC
  • They installed all Minerva and MINOS Software and have some problems with the ValidationTools, GMBrowser and RunVetoHVMonitor
  • Other than these 3 everyhing working and they started their shadow shift.

ValidationTools

  • UROC_sw_manager.py fail to “make clean” and “make” package
  • I removed the ValidationTools folder and ran UROC_sw_manager again – It worked!

GMBrowser

  • GMBrowser crashes (not responding)

Attempts

  • Update ControlRoomTools/gmbrowser folder to HEAD
    • Not worked!
  • Remove gmbrowser folder and run UROC_sw_manager.py
    • Not worked!

After these failed attempts, I concluded that the problem seems to be related with ROOT version they are using.

  • Rick installed ROOT 5.34 by building and tried running GMBrowser
    • It worked!

RunVetoHVMonitor

  • Script fail to open GUI
  • Problem is the missing “-x” option from SSH Command, they locally modified the script.
  • I need to modify and upload the correct version to CVS

 

UROC configure_runcontrol.sh Error

Problem

  • When you try to run “source configure_runcontrol.sh”
    • ImportError: No module named wx

Attempts

  1. Tried to import wx module to python for testing
    • python
    • import wx
    • That works! You can import wx into python
  2. Removed and re-installed package: python-wxgtk2.8
    • sudo apt-get remove python-wxgtk2.8
    • sudo apt-get install python-wxgtk2.8
    • Still getting the same error when I try to run the configure_runcontrol.sh
  3. Script: configure_runcontrol.sh runs the python script
    • mnvruncontrol/frontend/RunControlConfiguration.py
    • Running that script directly using “python RunControlConfiguration.py” works!

Tufts UROC Run Control Connection Error

Possible Reason

  • Dead SSH Connections between runcontrol server.

Error

Warning: remote port forwarding failed for listen port 3012
Exception in thread Thread-1:
Traceback (most recent call last):
File “/usr/lib/python2.7/threading.py”, line 551, in __bootstrap_inner
self.run()
File “/home/minerva/mnvruncontrol/backend/Threads.py”, line 236, in run
method_info["method"](*args, **kwargs)
File “RunControl.py”, line 1413, in ConnectDAQ
success = self.PrepareSSHTunnels(ssh_user=ssh_user, remote_host=remote_host, remote_port=remote_port)
File “RunControl.py”, line 1621, in PrepareSSHTunnels
deliveries = self.postoffice.SendTo(message=test_msg, recipient_list=recipients, timeout=3.0, with_exception=True)
File “/home/minerva/mnvruncontrol/backend/PostOffice.py”, line 1298, in SendTo
responses = self.SendWithConfirmation(message, timeout, with_exception)
File “/home/minerva/mnvruncontrol/backend/PostOffice.py”, line 1325, in SendWithConfirmation
raise AlreadyWaitingError(“SendWithConfirmation can’t wait for multiple messages simultaneously.”)
AlreadyWaitingError: SendWithConfirmation can’t wait for multiple messages simultaneously.

Attempts

  1. Restarted the computer
    • It did not worked!
  2. Tried to remove all SSH connections
    • ps -ef | grep ssh
    • kill <pID>
    • There was no idle SSH Connections
  3. Checked Tufts UROC-02 listener port
    • It was 3012 (same as UROC)
    • Changed UROC listener port to 3017
    • It worked!
  • Updated the UROC_User_Details wiki page for the new listener ports

UROC Update

  • Carrie had to do an emergency computer swap underground this afternoon.
  • We are no longer using mnvonline1 to read out the DAQ, we are using mnvonline0
  • UROCs must connect to  mnvonline1.fnal.gov instead of mnvonline0.fnal.gov
  • Carrie updated “mnvdaqrunscripts/def_mnvonline” and tagged it as carrie_2014_07_03
  • Updated UROC_sw_manager.py and informed all UROC users to update their UROCs.
    • Important Notice: After the update users need to remove the following folder
      • mnvruncontrol/backend/PostOffice

IP Change Problem

  • Message from Nathaniel indicating a problem with their networking infrastructure
    • “Reverse-IP lookup on our computer now returns a completely different IP address.  I have no idea what’s causing it.”
    • “SSH-ing directly into our nominal 205.133.226.69 address works fine… but our outbound connections look like they’re coming from 199.18.112.252!”
  • Otterbein University IT Department investigating the incident

UROC: Otterbein

  • RunControl Version Problem fixed by using older tag: “oaltinok_2014_04_11”
  • Updated “UROC_User_Details” for Otterbein University

Connection Problem

  • Otterbein UROC had problems connecting to all of the mnvonline machines.
  • Jeremy and I investigated the problem and find out the following
    • The Otterbein UROC hostname(photon.otterbein.edu) was not resolved from their Static IP address(205.133.226.69)
    • We added their Static IP address in addition to their hostname to the etc/hosts.allow
    • I suspect it is because they are using Ubuntu 14.04
  • Problem resolved

UROC Documentation

  • UROC information and documentation updated to an Online Version under Minerva Wiki
    • OLD MINERvA Document 5682
    • NEW https://cdcvs.fnal.gov/redmine/projects/minerva-ops/wiki/Setup_New_UROC
  • Plan is to update all UROC related documentation to Online

UROC: Otterbein

  • Modified UROC_user_details.xls
  • Included Special Kerberos Principal on required machines
  • E-mailed Carrie for modification of etc/hosts.allow
  • New Service ticket for ACL modification
  • mnvdaqrunscripts/install.sh modified with Otterbein hostname
  • mnvruncontrol/port_assignments.txt assigned new poart for Otterbein
  • UROC_sw_Manager.py Updated but it caused a problem due to a conflict on latest version of mnvruncontrol
  • Contacted with Jeremy, he will commit the lastest working version of runcontrol and let me know.

UROC: Tufts

Backup Disk

  • Seagate Expansion 2TB Desktop External Hard Drive USB 3.0 (STBV2000100)
  • Formatted disks with ext3 with Disc Names:
    • UROC01-BACKUP
    • UROC02
  • Created a folder named AutoBackup
  • Started first backup manually
  •  Set Auto Backup for semiweekly

UROC and Control Room Update

  • Heather and Joel has a new GUI for Veto Wall HV Monitoring
  • I have written a script: RunVetoHVMonitor
    • Script is added to Tools/ControlRoomTools/launchers and Tools/ControlRoomTools/linstall.sh
    • Tagged the version as oaltinok_2014_05_13
    • UROC_sw_manager.py updated to get the latest version of ControlRoomTools
  • Send an e-mail to UROC_LIST to inform users to update their UROCs
  • Updated the Control Room Computers: minerva-bm, -evd,-om-02, -rc
  • There were some local modifications on minerva-rc
    • MakeDST
    • gmbrowser/macros/OccupancyPlots.C
  • Sent and e-mail to inform Control Room People to know the updates

UROC and Control Room Update

  • Phil Rodrigues updated GMBrowser

    • GMBrowser to make it skip empty plots when it’s cycling. This way, it doesn’t spend most of the time showing the shifter empty canvases.
  • Version Tag: rodriges_2014-05-01
  • Updated the UROC_sw_manager.py with the tag
  • Committed to CVS and tagged UROC_sw_manager.py as CURRENT
  • Tested update on Tufts UROC and UROC02
  • Sent and e-mail to UROC-LIST to let people know the update
  • Minerba also updated Control Room Computers

UROC: Deborah Harris

MINERvA Software Authorization

  • Fixed .k5login entry for special kerberos principal. (Forgot to add principal with @FNAL.GOV)

MINOS RMS Insallation

  • Installed the version we use at Tufts – Contains modified rms.sh
  • Configurations as in wiki, used port 22
  • 2 Missing Libraries Installed
    • libjpeg62
    • libungif-bin
  • Fixed .k5login on minos-gateway-nd.fnal.gov with connecting using Tufts UROC. (Forgot to add principal with @FNAL.GOV)\
  • After some connection trials with different ports I started to get the following ERROR on Debbie’s and Tufts Uroc
    • Socket Server = localhost:19123
    • channel 2: open failed: connect failed: Connection refused
    • HistoDisplay: ERROR: Socket->Receive failed. resRecv = -1.
    • ==> Closing the socket connection..
    •  ==> Please open again.
    • Failed to get ConsumerInfo

UROC: Deborah Harris

UBUNTU 12.04 32-Bit Installation

  • Username: minerva
  • Hostname: 23-25-1-49-static.hfc.comcastbusiness.net
  • All Required software and ROOT 5.34 Installed
  • .bashrc modified for ROOT and Minerva Software Requirements
  • Kerberos installed and modified for FNAL.GOV connection

UROC Software Installation

  • All Software installed, UROC_sw_manager.py works!
  • Waiting for modification of the access control list (ACL) of the MINERvA NuMI Router from Fermilab IT Department

UROC: Deborah Harris

Bios Settings

  • Advanced
    • USB Configuration
      • USB Ports
        • Enable
      • USB Legacy
        • Enable
  • Boot
    • Boot Menu Type
      • Advance
    • Boot to Optical Devices
      • Disable
    • Boot to Removable Devices
      • Enable
    • Boot to Network
      • Disable
    • USB Boot
      • Enable
    • Boot USB Devices First
      • Enable
    • USB Mass Storage Emulation Type
      • All Fixed Disc

Authorization

  • Kerberos Special Principal

    • minerva-roc/minerva/23-25-1-49-static.hfc.comcastbusiness.net
  • Hostname
    • 23-25-1-49-static.hfc.comcastbusiness.net
  • Kerberos Special Principal added to the .k5login File on the following machines
    • mnvonlinemaster
    • mnvonline0
    • mnvonline1
    • mnvonline2
    • mnvonlinebck1
    • mnvonlinelogger
    • minerva-evd
  • Carrie McGivern added the Hostname to the /etc/hosts.allow on the same list above
    • /etc/hosts.allow modification requires root password
    • minerva-evd.fnal.gov /etc/hosts.allow Allows all hosts
  • ACNET
    • Contacted with Jim Smedinghoff (smedinghoff@fnal.gov)
    • minerva does not using ACNET anymore
  • MINOS
    • Contacted with Arthur E Kreymer (kreymer@fnal.gov)

Documentation

  • Uroc User Details.xls Modified with new UROC Information
  • Manual for Expert Authorization updated

 

UROC: Deborah Harris

  • Deborah Harris wants to use an old UROC at her home
  • Current Configuration
    • Static IP: 23.25.1.49
    • IP Host: 23-25-1-49-static.hfc.comcastbusiness.net
    • Machine Hostname: uroc.hep.utexas.edu
  • Static IP configured successfully and ping works
    • ping 23.25.1.49
    • ping 23-25-1-49-static.hfc.comcastbusiness.net
  • Package requires hostname check: Online/mnvdaqrunscripts/
    • install.sh modified: Added “23-25-1-49-static.hfc.comcastbusiness.net” as known hosts
    • tagged Online/mnvdaqrunscripts/ as v3r6p1
  • ControlRoomTools/UROC_sw_manager.py modified
    • Changed Online/mnvdaqrunscripts/ version to v3r6p1
  • Updated UROC User Details and uploaded to MINERvA-doc-9303