The aim of this project is to design and develop algorithms and architectures to construct autonomous mixed-initiative human-robot interactive systems. To achieve this, we integrate multiple physiological signal measurements and situational contextual information to assess different individual and team cognitive states such as workload, mind wandering, distraction, interference, and sense of urgency. We work on the examination of classified systemic human cognitive states in real-time and integrating them into a shared mental model which utilizes advanced probabilistic “theory of mind” paradigm. This can be further used to predict the team states of an autonomous human-robot interactive systems and support the adaptation between human and artificial teammates to achieve a shared goal. This project has been funded by AFORS.

Related Papers:

Scheutz, Matthias, Shuchin Aeron, Ayca Aygun, J. P. de Ruiter, Sergio Fantini, Cristianne Fernandez, Zachary Haga, Thuan Nguyen, and Boyang Lyu. “Estimating Systemic Cognitive States from a Mixture of Physiological and Brain Signals.” Topics in Cognitive Science (2023).

Ayca Aygun, Thuan Nguyen, Zachary Haga, Shuchin Aeron, Matthias Scheutz, “Investigating Methods for Cognitive Workload Estimation for Assistive Robots”, Sensors 22, no. 18, 2022: 6834.

Ayca Aygun, Boyang Lyu, Thuan Nguyen, Shuchin Aeron, Matthias Scheutz, “Cognitive Workload Assessment via Eye Gaze and EEG in an Interactive Multi-Modal Driving Task”, In Proceedings of the 2022 International Conference on Multimodal Interaction (ICMI), pp. 337-348, 2022.

In this project, we explore the effect of learning sign language on human’s cognitive workload based on a sign language tutoring game between a human and a humanoid robot. The game includes several words chosen from American Sign Language (ASL) which are imitated by the robot using several head gestures and body movements. During the experiment, similar words and gestures are imitated by the robot to intensify participant’s mental load given that distinguishing similar words/gestures require more attention. To assess the cognitive workload, the hand and body gestures of the participant, the response rates, the percentage of correct answers, and the response duration are examined. The results indicate that challenging questions, which are taken from similar words and gestures, result in increased number of hand and body gestures and decreased response performance while the response times are directly correlated with the level of difficulty. Moreover, the imitation of gestures with the robot have a positive effect on memorizing the signs.

Related Papers:

Ayca Aygun, “Assessment of Cognitive Load from Robot Assisted Sign Language Training”, 2021.

In this project, we introduce a robust method for accurate prediction of physiological parameters including interbeat interval (IBI) and heart rate variability (HRV) from cardiac signals acquired with wearable sensors in the presence of motion artifacts. The IBI and HRV assessment techniques, which is used for cardiac monitoring in free-living environments, can be highly affected by motion artifacts due to physical exercise. Therefore, it is essential to develop robust prediction techniques to be embedded into wearable sensors and used in daily life situations. The proposed approach includes two algorithmic components. First, we propose a combinatorial technique to select characteristic points that define heartbeats in noisy signals in time domain. The heartbeat detection problem is defined as a shortest path search problem on a direct acyclic graph that leverages morphological features of the cardiac signals by taking advantage of the time-continuity of heartbeats – each heartbeat ends with the starting point of the next heartbeat. The graph is constructed with vertices and edges representing candidate morphological features and IBIs, respectively. Second, we propose a fusion technique to combine physiological parameters estimated from different morphological features using the shortest path algorithm to obtain more accurate IBI/HRV estimations. We evaluate our techniques on motion-corrupted photoplethysmogram (PPG) and electrocardiogram (ECG) signals. This project was funded by National Institutes of Health (NIH), the National Science Foundation (NSF), and The Precise Advanced Technologies and Health Systems for Underserved Populations (PATHS-UP).

Related Papers:

Ayca Aygun, Hassan Ghasemzadeh, Roozbeh Jafari. “Robust Interbeat Interval and Heart Rate Variability Estimation Method from Various Morphological Features using Wearable Sensors”, IEEE Journal of Biomedical and Health Informatics (J-BHI), 2019. Acceptance Rate: 17.6%

Ayca Aygun, Roozbeh Jafari. “Robust Heart Rate Variability and Interbeat Interval Detection Algorithm in the Presence of Motion Artifacts”, IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), May 19-22, 2019, Chicago, IL, USA. Acceptance Rate: 14% (oral presentation)