Jonathan Rooney and Serena Thoma
The Blake-Perlman Computation and Collaboration Laboratory at Tufts University is a wonderful space where mechanical engineering students can work on their projects, collaborate in groups, and store their project. The space is equipped with tables, chairs, computers loaded with different engineering software, a printer, and many whiteboard surfaces. Although Blake is an ideal workspace for students, like any public space, it is beneficial for a student to know if there is a seat available before they come to Blake to work. Our goal for this project was to be able to detect the number of available seats in Blake at any given time and display the information on the Marauders’ Map.
With our knowledge of the layout of Blake, we decided to approach our solution using image processing. Most of the surfaces in Blake are white or light gray, meaning that we could use cameras that were already available to detect people and their belongings in contrast to the surroundings. With this approach in mind, we had to analyze the middle tables and computer tables with different methods. Each area of the room was assigned a camera and myRIO, allowing the system to continuously update the image it was analyzing, process the image, and send it to the data collection system of the Marauders’ Map.
Physical Set Up:
The physical monitoring setup consisted of a series of networked camera units connected to each other over the cloud. Each table was watched by a self-contained monitoring assembly, consisting of a USB webcam clamped to the ceiling framework and attached to a myRIO processor. The myRIO was concealed in a ceiling vent, along with a battery pack to power the unit.
With four tables to monitor, there were four units overall. Each unit ran a piece of code to analyze its individual table and output the number of available seats at that table. One central unit ran a second piece of code that compiled the data from its own table and from the other three units, which uploaded their data to Thingworx for the central unit to read. The central unit then pushed the overall number of available seats to Thingworx for the map itself to read.
In order to acquire images from the webcam attached to the myRIO, a project found at __ was altered to connect the camera images to the image processing. This project contained both a startup .vi to be run on the computer as well as a main .vi to be run on the myRIO. For the purposes of this project, the front panel was mainly used to debug whether or not the camera was connected properly, as the image indicator would update its images in real time.
Once the image was in the program, the LabView Vision Assistant was used to process the image. First the image was converted to a binary (black and white) image, then a threshold was applied to the image to find the brightest objects. For the images of Blake being analyzed, the brightest objects in the image are always the surfaces of the white tables, and any person occupying a seat at the table covers a certain area of the tabletop with their belongings. Once the table surfaces were isolated, the image was eroded and diluted to remove any particles not connected to the table, then the area of the table surface was analyzed and exported to the main .vi.
Images being analyzed
Back in the RT myRIO .vi, the current unoccupied area of the tabletop was used to calculate the percentage of table space that was unoccupied. Assuming that there were 10 seats at each table, the percentage was converted to a number of free seats, rounded to an integer, and sent to Thingworx in a unique property referring to the name on the myRIO in a “thing” called “Blake_Availability.”
The main .vi also contained a second Vision Assistant, which was used to detect any objects in the image that were darker than the floor and table. Although this image processor analyzed the image similarly as the bright object detection, the subvi outputted the number of particles it detected rather than the area of the particles. In the main .vi, this value was constantly compared to its previous value using a shift register, and would subtract one from the final free seat count if there was a change. This parameter was to detect people moving around the table, even if they were not sitting yet, notifying the user of people occupying space within Blake.
The computer tables were monitored in much the same way as the central tables, with images acquired via the same method. The way the seat count was performed, however, was significatly altered. Rather than analyzing the entire image, the software only monitored a region of interest between the base of the computer monitors and the front edge of the table. The image was converted to greyscale and then thresholded to create a binary image, which was run through several steps to eliminate small blobs.
Because the tables are white and the keyboards are black, the resulting image, when the table was empty, contained a single object wherever empty table space was present. If a person’s body was blocking the camera’s view of the section of table in front of a computer, there would be two table objects with a gap between them. If a second person was present, there would be three table objects, and so on. Therefore, the software performed a simple object detection to determine the number of occupants at computers. That number was then pushed to Thingworx for the central unit to receive.
In order to bring together the information acquired from all four myRIOs, a separate .vi was created to run on a desktop computer. This .vi collects the number of free seats sent from each individual myRIO and adds the numbers of free seats from the two middle table cameras and the two numbers from the two computer table cameras, creating only two integers had to be sent to the final map interface. The code then creates a string that defines the integers as two variables, “FreeSeats” and “FreeComputers,” in JSON format that can be understood by the map interface. Although this .vi does display data on a front panel, it was mostly used for debugging and therefore little attention was given to its aesthetics as the user is meant to view the data through the Marauders’ Map website.
Communication: Front Panel Communication: Block Diagram