Fermilab Power Failure

  • On Sunday 03:30 am there was a power failure affecting MINOS and MINERvA underground machines
  • Control Room Computers lost network mount to /minerva/data/
    • GMBrowser needs /minerva/data mounted and it was not working
    • minerva-evd is used by UROCs to mount /minerva/data and they are also affected.
    • Carrie opened a service ticket to ask Computer Division Help for Control Room Computers
    • Computer Division solved the incident and all machines and UROCs working properly.
  • mnvonlinebck1.fnal.gov machine is still down and we have no access to Veto Wall HV Monitoring.
  • e-Checklist can be used either one of the following servers: (minerva-wbm was down due to power glitch)
    • http://minerva-wbm.fnal.gov/minerva/echecklist/mininfo.php
    • http://nusoft.fnal.gov/minerva/echecklist/mininfo.php

minerva-om Update

  • minerva-om no longer support MINERvA Software
  • minerva-om has latest MINOS RMS Installed for om near check
  • .bashrc script modified to prompt users to use the “start_MinosOm.sh” command to start the minos om GUI
  • Nothing removed and every file and software are recoverable.


  • University of Minnesota Duluth group updating a pretty old UROC
  • They installed all Minerva and MINOS Software and have some problems with the ValidationTools, GMBrowser and RunVetoHVMonitor
  • Other than these 3 everyhing working and they started their shadow shift.


  • UROC_sw_manager.py fail to “make clean” and “make” package
  • I removed the ValidationTools folder and ran UROC_sw_manager again – It worked!


  • GMBrowser crashes (not responding)


  • Update ControlRoomTools/gmbrowser folder to HEAD
    • Not worked!
  • Remove gmbrowser folder and run UROC_sw_manager.py
    • Not worked!

After these failed attempts, I concluded that the problem seems to be related with ROOT version they are using.

  • Rick installed ROOT 5.34 by building and tried running GMBrowser
    • It worked!


  • Script fail to open GUI
  • Problem is the missing “-x” option from SSH Command, they locally modified the script.
  • I need to modify and upload the correct version to CVS


Status Update: Fake Pi0 Reconstruction

  • First we suspected it is due to proton hits
  • I did some tests on Cluster History at different stages of reconstruction
    • Before Everything
    • Before Muon
    • Before Michel
    • Before Proton
    • Before Pi0 Reconstruction
    • etc…
  • Find out that Pi0 Reconstruction DOES NOT touch proton prongs.
    • This is good. We are not having extra hits from proton/pion hits
    • However, this means that our first assumption is wrong! There is some other problem!
  • Prof. Mann and I decided that the best way to figure out what is going on is a scanning session of two sample of events
    • Signal with FAILED Reconstruction
    • Background with SUCCESSFUL Reconstruction

Cluster History Test

Test 1: Proton Prong Cluster History

  • Checking Cluster History of a single prong
    • Before: Used
    • After: Used

Test 2: Cluster History in Different Stages

  • Got all clusters and checked their history before each stage
    • Example Result:
    • N(Unused) N(Used)
    • Before Reco 41 120
    • Before Muon 41 120
    • Before Michel 41 120
    • Before Proton 41 120
    • Before Pi0 41 120
    • After Reco 0 161
  • Findings
    • Prong Clusters are already USED
    • There are additional clusters which are unused are used in Pi0 Reconstruction
    • In 2 sample Michel Stage also changed cluster History – See full list: NClusters Sheet1


  • Modified for CCProtonPi0 v1_04 Output
  • All Histograms Types Changed
    • Float to Double
    • TH1F to TH1D
    • TH2F to TH2D
  • Cut Table Statistics
    • Cut_Other: Represents all other cuts which are not tracked by CCProtonPi0 Package
  • Stacked Histogram for Pi0 Invariant Mass


  • Revised Pi0 Reconstruction
    • Global Variables Used in all functions
    • Removed unused functions and variables
    • ConeBlobs()
      • Variable Naming Match
      • Return type changed: StatusCode to bool
        • Returns false if setPi0ParticleData() fails
      • ConeBlobs() main function that controls Pi0 Reconstruction.
        • If it fails, the reconstructEvent() for that event stops.
    • VtxBlob()
      • Return type changed: StatusCode to bool
        • Always returns true (return type reserved for future implementation)
    • processBlobs()
      • Removed unused variables
      • Return type changed: StatusCode to void
  • Options File Modifications
    • New Options files for DEBUG
    • Original options file set to INFO


  • correctProtonProngEnergy()
    • Fixed a bug causing P4(Proton) = (nan,nan,nan,nan)
    • returns bool if Energy Correction Fails
  • setProtonParticleData()
    • double vertexZ no longer input parameter
    • Using Global Variable m_PrimaryVertex

Bug: correctProtonProngEnergy()


  • For some events 4-Momentum is zero
  • two fail modes observed:
    1. kinked tracks – if kinked track has 0 energy
    2. If particle already has 0 energy


  1. kinked tracks
    • Currently correctProtonProngEnergy() returns the final tracks energy as prong energy
    • Total Energy should be used
    • Summing 4-Momentum for each track in a prong
  2. If particle already has 0 energy
    • Check Particle energy before correctProtonProngEnergy()
    • if(E == 0) proton_isRecoGood = -1
    • else proton_isRecoGood = 1
  3. correctProtonProngEnergy() returns bool
    • returns false incase function fails
    • in this case use default particle 4-Momentum

UROC configure_runcontrol.sh Error


  • When you try to run “source configure_runcontrol.sh”
    • ImportError: No module named wx


  1. Tried to import wx module to python for testing
    • python
    • import wx
    • That works! You can import wx into python
  2. Removed and re-installed package: python-wxgtk2.8
    • sudo apt-get remove python-wxgtk2.8
    • sudo apt-get install python-wxgtk2.8
    • Still getting the same error when I try to run the configure_runcontrol.sh
  3. Script: configure_runcontrol.sh runs the python script
    • mnvruncontrol/frontend/RunControlConfiguration.py
    • Running that script directly using “python RunControlConfiguration.py” works!

Tufts UROC Run Control Connection Error

Possible Reason

  • Dead SSH Connections between runcontrol server.


Warning: remote port forwarding failed for listen port 3012
Exception in thread Thread-1:
Traceback (most recent call last):
File “/usr/lib/python2.7/threading.py”, line 551, in __bootstrap_inner
File “/home/minerva/mnvruncontrol/backend/Threads.py”, line 236, in run
method_info[“method”](*args, **kwargs)
File “RunControl.py”, line 1413, in ConnectDAQ
success = self.PrepareSSHTunnels(ssh_user=ssh_user, remote_host=remote_host, remote_port=remote_port)
File “RunControl.py”, line 1621, in PrepareSSHTunnels
deliveries = self.postoffice.SendTo(message=test_msg, recipient_list=recipients, timeout=3.0, with_exception=True)
File “/home/minerva/mnvruncontrol/backend/PostOffice.py”, line 1298, in SendTo
responses = self.SendWithConfirmation(message, timeout, with_exception)
File “/home/minerva/mnvruncontrol/backend/PostOffice.py”, line 1325, in SendWithConfirmation
raise AlreadyWaitingError(“SendWithConfirmation can’t wait for multiple messages simultaneously.”)
AlreadyWaitingError: SendWithConfirmation can’t wait for multiple messages simultaneously.


  1. Restarted the computer
    • It did not worked!
  2. Tried to remove all SSH connections
    • ps -ef | grep ssh
    • kill <pID>
    • There was no idle SSH Connections
  3. Checked Tufts UROC-02 listener port
    • It was 3012 (same as UROC)
    • Changed UROC listener port to 3017
    • It worked!
  • Updated the UROC_User_Details wiki page for the new listener ports


  • TG4Trajectory Map Removed
  • Muon Reconstruction
    • Using Global Variables
      • m_MuonTrack
      • m_MuonProng
      • m_MuonParticle
    • SetMuonParticleData() Improved
  • Proton Reconstruction
    • Fixed a bug causing empty NTuple Branches even with a successful proton reconstruction
    • Using Global Variables
      • m_ProtonProngs
      • m_ProtonParticles
    • Stop Algorithm if CreateTrackedParticles() fails
    • New Variable Cut_Particle_None
    • Removed Proton Score Cut
    • getProtonProng() Improved
    • SetProtonParticleData() Improved
    • Branches initialized to SENTINEL = -9.9
  • Global Variable: m_PrimaryVertex
    • Still need to pass the variable to CCPi0 Functions – Some of the functions outside the Global Scope
  • Found a new bug in correctProtonProngEnergy()
    • If the function changes proton energy, NTuple Branches for Momentum and Energy does not filled correctly
    • Will fix this with v1_03

Debugging for Proton Reconstruction

Problem and Reason

  • Although recontructEvent() returns with a successful proton reconstruction; in some cases the NTuple branches for proton are empty.
  • Possible Reason:
    • getProtonProng() and setProtonParticleData() functions are not correctly working!
  • Using Global Variables making sure that getProtonProng() and setProtonParticleData() functions access SAME proton prong information.
  • This fix will be included under v1_2

Test Details

Test 1:

  • Global Variable:
    • testDouble
  • Function Used:
    • getProtonProng()
  • Did not passed variables to function
  • Assigned Value:
    • Before = 1987
    • Inside = 1986
  • Observed Value:
    • Before Call = 1987
    • Inside, Before Assignment = 1987
    • Inside, After Assignment = 1986
    • After Call = 1986
  • Test Results:
    • Do not PASS Global Variables to the Functions

Test 2:

  • Class Private Member:
    • testDouble
  • Test Results:
    • DOES NOT Compile
    • const functions CANNOT modify member variables

Test 3:

  • Global Variable:
    • testDouble
  • Functions Used:
    • getProtonProng() and setProtonParticleData()
  • Did not passed variables to function
  • Assigned Values and Observed Values were matched
  • Test Results:
    • getProtonProng() and setProtonParticleData() can access and modify the testDouble

Test 4:

  • Global Variables:
    • Minerva::ProngVect m_protonProngs;
    • Minerva::ParticleVect m_protonParticles;
  • Function Used: getProtonProng()
    • Before Call, m_protonProngs.size() = 0
    • After Call, m_protonProngs.size() = 1
  • Did not passed variables to functions
  • Test Results:
    • getProtonProng() can access and modify m_protonProngs;

Test 5:

  • Global Variables:
    • Minerva::ProngVect m_protonProngs;
    • Minerva::ParticleVect m_protonParticles;
  • Function Used:
    • getProtonProng() and setProtonParticleData()
  • Did not passed variables to functions
  • Test Results:
    • getProtonProng() and setProtonParticleData() can access and modify m_protonProngs;
    • PROBLEM: Need to clear m_protonProngs vector before each iteration.

Test 6:

  • Global Variables:
    • Minerva::ProngVect m_protonProngs;
    • Minerva::ParticleVect m_protonParticles;
  • Function Used:
    • getProtonProng() and setProtonParticleData()
  • Did not passed variables to functions
  • Testing for actual data analysis
  • Test Results:
    • Successful! – Works as expected!

Nearonline Dispatcher Crash

  • Yesterday around 15:00, nearonline dispatcher crashed causing no raw data under: /scratch/nearonline/var/job_dump/
  • Manual Job Submission via manual_dst_submit.py did not worked. (No Raw Data)
  • Jeremy manually copied the raw data with the following command and manually submitted the missing jobs:
    • cd /scratch/nearonline/var/job_dump/
    • for ((i=22;i<=32;i++)); do cp /mnvonline0/work/data/rawdata/MV_00010504_00${i}_*_RawData.dat .; done


  • New Histograms for preFilter() Results
  • Pi0 invariant mass histogram moved under Pion Class
  • Studies integrated with package
    • Studies folder removed
  • Proton Reconstruction fails even foundProton triggerred as true.
    • I will investigate it on Monday

UROC Update

  • Carrie had to do an emergency computer swap underground this afternoon.
  • We are no longer using mnvonline1 to read out the DAQ, we are using mnvonline0
  • UROCs must connect to  mnvonline1.fnal.gov instead of mnvonline0.fnal.gov
  • Carrie updated “mnvdaqrunscripts/def_mnvonline” and tagged it as carrie_2014_07_03
  • Updated UROC_sw_manager.py and informed all UROC users to update their UROCs.
    • Important Notice: After the update users need to remove the following folder
      • mnvruncontrol/backend/PostOffice


  • PreFilterPi0()
    • Getting Vertex as input
    • Return Value Changed: StatusCode -> bool
    • Debug Messages and Comments
    • New Variables:
      • Rejected Energy
      • Status
        • preFilter_Result     
        • 0 = Passes Filter     
        • 1 = Rejected by Target Filter     
        • 2 = Rejected by Max Other     
        • 3 = Rejected by Min Other
  • tagTruth()
  • Final State Particle Table as INFO
    • m_writeFSParticle_Table controls execution
  • Using reverse_iterator
  • setParticleData Functions
    • Return Value Changed: StatusCode -> bool