Human-Robot Collaboration (HRC) is increasingly common in physically demanding industrial settings. However, the impact of variable human states, such as physical fatigue, on trust and the quality of this collaboration remains unclear. This is especially critical during HRC that entails co-lifting of industrial materials. This study examines the impact of operator physical fatigue on trust in a robotic partner, as well as associated perceptions of robot effort, coordination, lift quality, and objective lift performance during a co-lifting task that requires complex maneuvering. In a within-subjects experiment, 40 healthy adults (20 females) performed a collaborative lifting task through an asymmetrical complex lift trajectory under two conditions: no-fatigue and fatigue, induced via physical exertion of the dominant arm. We collected subjective ratings of trust, perceived exertion, lift quality, robot effort, and coordination, alongside objective performance metrics based on movement trajectories, such as similarity (using dynamic time warping), acceleration, and jerks. The results demonstrate that physical fatigue, as evidenced by increased perceived exertion ratings, significantly decreased participants' trust in the robot (particularly in the late phase), despite no change in the robot's actual performance. Interestingly, the impacts of fatigue on perceptions of effective coordination were dynamically impacted by the early (higher under fatigue) versus the late (lower under fatigue) phases; participants viewed the robot's effort as less consistent only in the late phase when fatigued. Surprisingly, while fatigue did not affect any of the objective performance metrics based on movement trajectories, participants perceived an improvement in transport quality; however, this was only found in the early phase. These findings indicate that physical fatigue alters not only the physical capacity of the operator but also crucial perceptual shifts of the robot partner and perceived transport quality. This highlights the need for more objective metrics to evaluate HRC that are resilient to operator fatigue. Doing so may enable the development of robust HRC systems that ensure effective and trustworthy collaborations.

Trust in human–robot collaboration (HRC) is influenced by situational factors, such as human states of fatigue and robot performance (e.g., reliability). In industrial settings, operators are often fatigued which increases attentional demands. This sustained mental demand may potentially alter the trusting behaviors of the operator. Whether this happens, and why, are currently unexplored. While most studies have relied on self-reports, recent advancements in neuroergonomics have enabled the assessment of neural signatures of trust. We hypothesize that shedding light on the neural mechanisms during HRC, under varying trust states, may provide additional insights (beyond trust perceptions) into the subconscious cognitive or affective states associated with trust under cognitive fatigue. To test this hypothesis, we conducted an HRC experiment on 16 sex-balanced participants in which trust was manipulated via robot reliability at two sessions (fatigue session and no-fatigue session). We monitored brain activity, heart rate variability (HRV), performances, and subjective experiences to investigate the effect of cognitive fatigue and robot reliability on trust metrics in HRC. A significant main effect of cognitive fatigue and reliability was observed on brain activations, functional and effective connectivity, HRV, subjective response, and performance. While most measures were sensitive to changes in cognitive fatigue and reliability, only neural responses revealed interactions between reliability, cognitive fatigue, and sex. Anterior prefrontal cortex (APFC)–left dorsolateral prefrontal cortex (LDLPFC) connection strength decreased with unreliable robot behavior in non-fatigued states but unexpectedly increased when fatigue was present. This suggests a neural strategy shift involving greater top-down influence from executive to motivational regions to exert more mental effort in challenging HRC conditions. Insights from this investigation can advance objective trust measures under cognitive fatigue states and help design better collaborations.

This study examines the impact of proactive takeovers on driver trust and associated neural and gaze behaviors across varying automated vehicle (AV) performances. Fifty-seven participants (27 females and 30 males) were grouped by their takeover behaviors across four events that simulated AV capabilities of crash avoidance and silent failures. Participants reported subjective trust scores after experiencing the events. Functional near-infrared spectroscopy-based neural responses included peak frontal, motor, and visual brain activation. Eye tracking-based gaze measures included fixations and gaze entropy metrics. Results indicated that takeover behaviors were associated with lower trust ratings regardless of AV performance. Drivers who proactively took over exhibited suppressed brain activation across the dorsal–lateral prefrontal cortex (DLPFC), intermediate frontal cortex, supplementary motor area, and secondary/tertiary visual cortex, and this was accompanied by greater fixations on the AV disengagement control. On the other hand, drivers who did not take over rated higher trust and exhibited increased DLPFC activity along with greater fixations on the rear-view mirror. The neural and gaze measures, when combined with distrusting takeover behaviors, implicate (1) risk avoidance and bottom-up information processing (e.g., anticipatory gaze behavior) strategies adopted by drivers who demonstrated proactive takeover behaviors and (2) reliance on top–down cognitive processes and prior experiences to assess AV performances by drivers who did not take over regardless of AV performance. Along with underscoring the utility of these granular metrics for informing adaptive human–AV interactions, our study emphasizes the importance of integrating proactive takeover paradigms to examine perception–cognition–action processes of (dis)trusting behaviors during automated driving.

Augmented reality has shown promise as an assistive tool for mass casualty incident (MCI) triage to facilitate cognitive support during live response and to furnish immersive virtual MCI training. It has been shown that women and people of color experience less effective triaging in emergency room settings, yielding diminished healthcare outcomes. In the more chaotic and austere MCI event, triage efforts further determine which patients are left to expire and which get life-saving interventions before they arrive at the hospital. This work offers an exploratory within subjects investigation with actual first responders (N = 13) to examine preliminary patterns about whether diminished triage efficacy based on patient demographics can be captured during an augmented-reality virtual training simulation (1V) and/or within a 'real' patient training simulation (supported with AR triage interfaces) with patient actors (2R). While both conditions used a Hololens2, the 1V condition displayed AR virtual patients, whereas the 'patient actor' condition had real people scattered in a physical space needing triaged. We conduct ANOVAS and construct linear models to evaluate our data. Our exploratory analysis finds slight potential impacts of patient demographics in triage efficacy within the 'real' simulation, and more severe impacts in the virtual simulation. We further discuss implications of this work and explore underlying cognitive factors.

Augmented Reality (AR) training is a cost-effective and safe alternative to traditional instructional methods. However, training novices in basic mid-air AR interactions remains challenging. To address this, we aimed to: (a) develop a robust metric to evaluate user performance across different AR interaction techniques and develop adaptation models to predict additional training requirements; (b) evaluate the adaptation models using a neuroergonomics approach. We conduct a two-phase study during which, novice participants perform simple AR interactions: poking and raycasting. In Phase-I, twenty-seven participants’ data is used to identify a bi-variate performance metric based on median completion ime and consistency. Unsupervised models are trained using this metric to classify participants as low/high performers. In Phase-II, we evaluate the models on twenty-one new participants and analyze the differences in performance, neural activity and heart-rate variability between low/high performers. Our study showcases the effectiveness of our models and further discusses the potential of integrating neuroergonomics for advanced AR-based training applications.

Anomalous robot motions caused by cyber attacks and inherent defects can lead to task failures as well as harmful accidents in collaborative human-robot workplaces. External Internet-of-Things (IoT) sensors are essential for reliably monitoring robot tool paths and detecting deviations as early as possible, especially in anomalous situations where the robot system and its internal sensors may not function as expected. However, affordable external IoT sensors may suffer from poor-quality measurements, necessitating data-driven algorithmic enhancements. In addition to external sensors providing independent monitoring, to effectively detect trajectory deviations, we need anomaly detection methods that can capture complex dynamic patterns in the nonlinear trajectory time-series data. In this work, we propose a framework to accurately estimate robot trajectories during normal robot operations as well as to detect various types of anomalous trajectory changes using an external camera. The proposed framework efficiently incorporates marker-based pose estimation, Long Short-Term Memory (LSTM), and residual control charts. The framework was evaluated in a shared human-robot assembly task. The results show that we can accurately estimate robot trajectories by enhancing camera-based measurements. Moreover, it effectively detects anomalous trajectory changes in their early stages. The motion deviation upon detection assists in determining a safe working distance. The framework is also generalizable to previously unseen trajectory deviations and allows the use of a variety of IoT sensors.

Trust in human–robot collaboration is an essential consideration that relates to operator performance, utilization, and experience. While trust’s importance is understood, the state-of-the-art methods to study trust in automation, like surveys, drastically limit the types of insights that can be made. Improvements in measuring techniques can provide a granular understanding of influencers like robot reliability and their subsequent impact on human behavior and experience. This investigation quantifies the brain–behavior relationships associated with trust manipulation in shared space human–robot collaboration to advance the scope of metrics to study trust. Thirty-eight participants, balanced by sex, were recruited to perform an assembly task with a collaborative robot under reliable and unreliable robot conditions. Brain imaging, psychological and behavioral eye-tracking, quantitative and qualitative performance, and subjective experiences were monitored. Results from this investigation identify specific information processing and cognitive strategies that result in identified trust-related behaviors that were found to be sex specific. The use of covert measurements of trust can reveal insights that humans cannot consciously report, thus shedding light on processes systematically overlooked by subjective measures. Our findings connect a trust influencer (robot reliability) to upstream cognition and downstream human behavior and are enabled by the utilization of granular metrics.

This paper presents the development and validation of a robot adaptation model to support human operators in Human-Robot Collaborative tasks when they are cognitively fatigued. A human-centered robot adaptation method for providing appropriate assistance to the operator is developed with a dual objective of task performance optimization and aiding human cognitive fatigue recovery. The problem is formulated as a Markov Decision Process (MDP) and solved using Q-learning. The implementation issues resulting from modeling the MDP and performing Q-learning for cognitive fatigue recovery, methods to mitigate those issues, and implications on the resulting optimal policies are discussed. The proposed approach is evaluated through a user study of sixteen participants performing a robotic surface polishing task under cognitive fatigue conditions. The MDP model is validated using subjective metrics (fatigue perception surveys) and objective metrics (Heart-Rate Variability (HRV), accuracy in trajectory tracking, and time efficiency of the task). Fatigue perceptions, accuracy, and time efficiency improved during the user-specific optimal adaptation policies. HRV analysis of time-domain features shows an overall improvement in fatigue conditions during the optimal adaptation policies. The results from this approach indicate that such human-centered robot adaptation can lead to efficient human-robot collaborations with robust interactions between robots and humans.

The effects of cognitive stress are complex and multi-dimensional with nuanced neural and physiological representations across our lifespan. Chronic and instantaneous stressors are known to alter both executive function and motor performance — a particularly challenging prospect for older adults. Age, sex, and motor activity are critical yet under-represented dimensions in the domain of stress detection. Through the present work, we explore a subset of these variables and the relevance of brain hemodynamics and heart rate variability (HR/V) as biomarkers of stress in an aging population. We rely on a multimodal, sex-balanced, motor-stress data set (N = 59) and an exhaustive machine learning workflow to operationalize the unique neurophysiological states that form the human stress response. We found that a quadratic discriminant was sufficient to separate the two states across feature, demographic, and activity variables. We report a stress detection accuracy between$78-98\%$when using models trained independently on each feature-set. However, these models were highly sensitive to sex, and activity differences — with distinct regions, and features implicated in stress recognition. Both HR/V and fNIRS based features were excellent indices of cognitive stress, however neither generalized to a degree beneficial toward operational use. Our observations underscore the importance of task-context, age, and sex as factors in modeling stress detection tools for older adults.

In recent years, robots have become vital to achieving manufacturing competitiveness. Especially in industrial environments, a strong level of interaction is reached when humans and robots form a dynamic system that works together towards achieving a common goal or accomplishing a task. However, the human-robot collaboration can be cognitively demanding, potentially contributing to cognitive fatigue. Therefore, the consideration of cognitive fatigue becomes particularly important to ensure the efficiency and safety in the overall human-robot collaboration. Additionally, sex is an inevitable human factor that needs further investigation for machine learning model development given the perceptual and physiological differences between the sexes in responding to fatigue. As such, this study explored sex differences and labeling strategies in the development of machine learning models for cognitive fatigue detection. Sixteen participants, balanced by sex, recruited to perform a surface finishing task with a UR10 collaborative robot under fatigued and non-fatigued states. Fatigue perception and heart rate activity data collected throughout to create a dataset for cognitive fatigue detection. Equitable machine learning models developed based on perception (survey responses) and condition (fatigue manipulation). The labeling approach had a significant impact on the accuracy and F1-score, where perception-based labels lead to lower accuracy and F1-score for females likely due to sex differences in reporting of fatigue. Additionally, we observed a relationship between heart rate, algorithm type, and labeling approach, where heart rate was the most significant predictor for the two labeling approaches and for all the algorithms utilized. Understanding the implications of label type, algorithm type, and sex on the design of fatigue detection algorithms is essential to designing equitable fatigue-adaptive human-robot collaborations across the sexes.

In this work we present ARTTS: a head-worn Augmented Reality (AR) Triage Tool Suite containing an initial sorting tool, virtual assessment tool, and virtual triage tag to assist emergency responders in mass casualty incidents. The initial sorting tool can prompt novice responders through first-wave tasks to aid recalibration from shock to triage. The virtual assessment tool provides novice responders, potentially confused by the chaos, with a walkthrough of the SALT triage flowchart. Finally, current emergency medical triage processes leverage static paper tags susceptible to loss or illegible damage. ARTTS’ virtual triage tags are dynamic, can be updated through responder interaction, and employ user interface emergent features based on individual patient conditions. This paper describes ARTTS’ capabilities, as well as the applied user-centered design process including review of existing triage material, subject-matter expert interview transcripts, wireframing, application of usability and user-centered design principles, as well as iterative usability subject-matter expert assessments and design walkthroughs. The ARTTS user experience aims to enhance, not upend, existing triage processes. Finally, this paper provides a usability evaluation comparing ARTTS’ virtual triage tag to a physical paper triage tag. Our tag achieved requisite System Usability Scale (SUS) scores and showed negligible differences to the paper triage tag on usability and mental workload.

Vigilance is a primary job-performance requirement for human operators in domains that demand sustained attention, including air traffic control (ATC), surveillance, emergency response and many others. In this pilot study we introduce the prerequisites and conditions that facilitate a novel closed-loop, adaptive neurostimulation system to alleviate vigilance decrements during prolonged time-on-task efforts. Here, we investigate the use of transcranial Direct Current Stimulation (tDCS) with preset stimulation parameters - intensity (I), duration (t), and, probe location to augment the operators' vigilance state during an under-arousing task. To this end, we employ an app-based psychomotor-vigilance test (PVT), where performance metrics are analyzed along with physiological and cognitive bio-markers to explore opportunities toward a predictive framework. Initial observations (N = 19) suggest that - (1) a prolonged version of the PVT (40 min.) can function both as a diagnostic and an inductive mechanism for vigilance loss, (2) tDCS can serve to restore/ improve operator vigilance states relative to baseline performance levels, and (3) short-term heart rate variability (HR/V) features (3 min.) and the fNIRS signal are sensitive to state changes during the PVT, and to the effects of stimulation.

To quantify mental effort and track neurophysiological changes due to powered-robotic-exoskeleton based gait training objectively, we investigated the feasibility of using functional connectivity and neural efficiency metrics obtained from functional near infrared spectroscopy to monitor neurophysiological changes during powered robotic exoskeleton-based gait training in two stroke and two spinal cord injury (SCI) patients. Increased functional connectivity between different brain regions were associated with improved gait performance in stroke patients but indicated increased mental workload with no gait changes in SCI patients. Neural efficiency provided cost of maintaining motor performance in all four patients. Both metrics show potential in tracking mental effort and gait training progress and may serve as valuable inputs to rehab exoskeleton brain computer interfaces and contribute to the development of personalized rehabilitation programs.

In this paper, we analyze various kinds of hand tremors in the time and frequency domain, that are induced by performing a set of hand actions. We collected the tremor data using a simple, wearable accelerometer from 15 healthy individuals that had varying levels of athleticism. The overall results presented here show that the physiologic tremors in range of 8-14 Hz are most noticeable under fatigue.