Collaborative problem-solving under time pressure is common but difficult, as teams must generate ideas quickly, coordinate actions, and track progress. Generative AI offers new opportunities to assist, but we know little about how proactive agents affect the dynamics of real-time, co-located teamwork. We studied two forms of proactive support in digital escape rooms: a facilitator agent that offered summaries and group structures, and a peer agent that proposed ideas and answered queries. In a within-subjects study with 24 participants, we compared group performance and processes across three conditions: no AI, peer, and facilitator. Results show that the peer agent occasionally enhanced problem-solving by offering timely hints and memory support; however, it also disrupted flow, increased workload, and created over-reliance. In comparison, the facilitator agent provided light scaffolding but had a limited impact on outcomes. We provide design considerations for proactive generative AI agents based on our findings.

As robots and autonomous agents integrate into society, understanding their influence on human social dynamics is crucial. We investigate human–robot interactions, focusing on the impact of prosocial behavior by robots on subsequent human interactions and humans’ willingness to exhibit prosocial behavior toward robots. Our study involved a token-collection game in a grid-world environment. Players, human or robot, could become trapped; a prosocial action involved another player freeing the trapped individual. Findings indicate that robots demonstrating prosocial behavior toward humans can inspire prosocial behavior toward others. Humans also show a notable propensity to assist robots. Witnessing robots engage in prosocial behavior may activate social norms related to cooperation, prompting humans to emulate these behaviors. Robots’ actions could improve the saliency of these acts, focusing people’s attention on prosocial behaviors they might not notice otherwise. Overall, the findings suggest that robots can promote prosocial behavior among humans, contributing to a more cooperative social environment. This research has implications for design and implementation of future autonomous systems, emphasizing the importance of social considerations in human-AI interaction studies.

Human state detection and behavior prediction have seen significant advancements with the rise of machine learning and multimodal sensing technologies. However, predicting prosocial behavior intentions in mobility scenarios, such as helping others on the road, is an underexplored area. Current research faces a major limitation—there are no large, labeled datasets available for prosocial behavior, and small-scale datasets make it difficult to train deep-learning models effectively. To overcome this, we propose a self-supervised learning approach that harnesses multi-modal data from existing physiological and behavioral datasets. By pre-training our model on diverse tasks and fine-tuning it with a smaller, manually labeled prosocial behavior dataset, we significantly enhance its performance. This method addresses the data scarcity issue, providing a more effective benchmark for prosocial behavior prediction, and offering valuable insights for improving intelligent vehicle systems and human-machine interaction.

For future human-autonomous vehicle (AV) interactions to be effective and smooth, human-aware systems that analyze and align human needs with automation decisions are essential. Achieving this requires systems that account for human cognitive states. We present a novel computational model in the form of a Dynamic Bayesian Network (DBN) that infers the cognitive states of both AV users and other road users, integrating this information into the AV's decision-making process. Specifically, our model captures the ``well-being'' of both an AV user and an interacting road user as cognitive states alongside trust. Our DBN models infer beliefs over the AV user’s evolving well-being, trust, and intention states, as well as the possible well-being of other road users, based on observed interaction experiences. Using data collected from an interaction study, we refine the model parameters and empirically assess its performance. Finally, we extend our model into a causal inference model (CIM) framework for AV decision-making, enabling the AV to enhance user well-being and trust while balancing these factors with its own operational costs and the well-being of interacting road users. Our evaluation demonstrates the model’s effectiveness in accurately predicting user's states and guiding informed, human-centered AV decisions.

In hybrid mobility societies, where automated vehicles (AVs) and humans interact in public spaces, the significance of prosocial behaviors intensifies. These behaviors are crucial for the smooth functioning of an interdependent transportation environment, mitigating challenges from the integration of AVs and human-operated systems, and enhancing user well-being by fostering more efficient, less stressful, and inclusive environments. This study explores the impact of receiving prosocial behaviors on cognition, riding behavior, and well-being of micromobility users through interdependent traffic situations within a simulated urban environment. Our mixed design study involved two types of social interactions as between-subject conditions of prosocial and asocial interaction, and three categories of time constraint as within-subject conditions: relaxed, neutral, and pressed. The findings reveal that receiving prosocial and asocial behaviors can affect the state of well-being and trial performance in a mobility environment.

Micromobility devices, such as e-scooters and delivery robots, hold promise for eco-friendly and cost-effective alternatives for future urban transportation. However, their lack of societal acceptance remains a challenge. Therefore, we must consider ways to promote prosocial behavior in micromobility interactions. We investigate how post-ride feedback can encourage the prosocial behavior of e-scooter riders while interacting with sidewalk users, including pedestrians and delivery robots. Using a web-based platform, we measure the prosocial behavior of e-scooter riders. Results found that post-ride feedback can successfully promote prosocial behavior, and objective measures indicated better gap behavior, lower speeds at interaction, and longer stopping time around other sidewalk actors. The findings of this study demonstrate the efficacy of post-ride feedback and provide a step toward designing methodologies to improve the prosocial behavior of mobility users.

Humans’ internal states play a key role in human-machine interaction, leading to the rise of human state estimation as a prominent field. Compared to swift state changes such as surprise and irritation, modeling gradual states like trust and satisfaction are further challenged by label sparsity: long time-series signals are usually associated with a single label, making it difficult to identify the critical span of state shifts. Windowing has been one widely-used technique to enable localized analysis of long time-series data. However, the performance of downstream models can be sensitive to the window size, and determining the optimal window size demands domain expertise and extensive search. To address this challenge, we propose a Selective Windowing Attention Network (SWAN), which employs window prompts and masked attention transformation to enable the selection of attended intervals with flexible lengths. We evaluate SWAN on the task of trust prediction on a new multimodal driving simulation dataset. Experiments show that SWAN significantly outperforms an existing empirical window selection baseline and neural network baselines including CNN-LSTM and Transformer. Furthermore, it shows robustness across a wide span of windowing ranges, compared to the traditional windowing approach.

We propose a framework for detecting user driving style preference with multimodal signals, to adapt autonomous vehicle driving style to drivers’ preferences in an automatic manner. Mismatch between the automated vehicle driving style and the driver’s preference can lead to more frequent takeovers or even disabling the automation features. We collected multi-modal data from 36 human participants on a driving simulator, including eye gaze, steering grip force, driving maneuvers, brake and throttle pedal inputs as well as foot distance from pedals, pupil diameter, galvanic skin response, heart rate, and situational drive context. Based on the data, we constructed a data-driven framework using convolutional Siamese neural networks (CSNNs) to identify preferred driving styles. The model performance has significant improvement compared to that in the existing literature. In addition, we demonstrated that the proposed framework can improve model performance without network training process using data from target users. This result validates the potential of online model adaption with continued driver-system interaction. We also perform an ablation study on sensing modalities and present the importance of each data channel.

Advanced driver assistance systems (ADAS) need to account for the driver’s awareness of the environment to be effectively used. This study examines the impact of environmental features (eg, visual complexity, object density, roadway type, lighting) on drivers’ situation awareness (SA). This is achieved using a controlled study with 40 participants. Using a split-plot design, the participants were shown 30 out of 75 real-world driving scenarios displayed in a driving simulator environment. Participants’ responses to Situational Awareness Global Assessment Technique (SAGAT) queries on the type and coordinates of objects in the scene were used to calculate SA scores. A hurdle model was developed to estimate participants’ SA scores. The key findings highlight visual complexity as a significant predictor of SA scores. This predictor was easy to compute and able to capture the complexity of objects that impact road safety as well as the visual clutter in the background. The model showed that drivers were able to identify at least one object of interest in complex environments with high visual complexity and with many objects. A higher proportion of vulnerable road users was associated with a greater likelihood of a non-zero SA score, but the SA score was lower compared to environments with higher proportions of cars. The findings of this study provide insights into the environmental factors to be considered for SA predictive models.

Unpaired video-to-video translation aims to translate videos between a source and a target domain without the need of paired training data, making it more feasible for real applications. Unfortunately, the translated videos generally suffer from temporal and semantic inconsistency. To address this, many existing works adopt spatiotemporal consistency constraints incorporating temporal information based on motion estimation. However, the inaccuracies in the estimation of motion deteriorate the quality of the guidance towards spatiotemporal consistency, which leads to unstable translation. In this work, we propose a novel paradigm that regularizes the spatiotemporal consistency by synthesizing motions in input videos with the generated optical flow instead of estimating them. Therefore, the synthetic motion can be applied in the regularization paradigm to keep motions consistent across domains without the risk of errors in motion estimation. Thereafter, we utilize our unsupervised recycle and unsupervised spatial loss, guided by the pseudo-supervision provided by the synthetic optical flow, to accurately enforce spatiotemporal consistency in both domains. Experiments show that our method is versatile in various scenarios and achieves state-of-the-art performance in generating temporally and semantically consistent videos. Code is available at: https://github.com/wangkaihong/Unsup_Recycle_GAN/.

As autonomous vehicles (AVs) become ubiquitous, users' trust will be critical for the successful adoption of such systems. Prior works have shown that the driving styles of AVs can impact how users trust and rely on such systems. However, users' preferred driving style may vary with changes in trust or road conditions, experience, and personal driving preferences. We explore methods to adapt the driving style of an AV to match the preferred driving style of users to improve their trust in the vehicle. We conducted a pilot study ($n=16$) on a simulated urban environment, where the users experience various static and adaptive driving styles for different pedestrian and traffic-related scenarios. Our results indicate that users best trust AVs that closely match their preferences ($p< 0.05$). We believe that exploring the effects of AV driving style on users' trust and workload will provide necessary steps towards developing human-aware automated systems.

Despite the recent advancement in driver assistance systems, most existing solutions and partial automation systems such as SAE Level 2 driving automation systems assume that the driver is in the loop; the human driver must continuously monitor the driving environment. Frequent transition of maneuver control is expected between the driver and the car while using such automation in difficult traffic conditions. In this work, we aim to predict driver takeover timing in order for the system to prepare transition from automation to driver control. While previous studies indicated that eye gaze is an important cue to predict driver takeover, we hypothesize that traffic condition as well as the reliability of the driving automation also have a strong impact. Therefore, we propose an algorithm that jointly consider the driver’s gaze information and contextual driving environment, which is complemented with the vehicle operational and driver physiological signals. Specifically, we consider joint embedding of traffic scene information and gaze behavior using 3DConvolutional Neural Network (3D-CNN). We demonstrate that our algorithm is successfully able to predict driver takeover intent, using user study data from 28 participants collected in simulated driving environments.

A key factor to optimal acceptance and comfort of automated vehicle features is the driving style. Mismatches between the automated and the driver preferred driving styles can make users take over more frequently or even disable the automation features. This work proposes identification of user driving style preference with multimodal signals, so the vehicle could match user preference in a continuous and automatic way. We conducted a driving simulator study with 36 participants and collected extensive multimodal data including behavioral, physiological, and situational data. This includes eye gaze, steering grip force, driving maneuvers, brake and throttle pedal inputs as well as foot distance from pedals, pupil diameter, galvanic skin response, heart rate, and situational drive context. Then, we built machine learning models to identify preferred driving styles, and confirmed that all modalities are important for the identification of user preference. This work paves the road for implicit adaptive driving styles on automated vehicles.

Although partially autonomous driving (AD) systems are already available in production vehicles, drivers are still required to maintain a sufficient level of situational awareness (SA) during driving. Previous studies have shown that providing information about the AD’s capability using user interfaces can improve the driver’s SA. However, displaying too much information increases the driver’s workload and can distract or overwhelm the driver. Therefore, to design an efficient user interface (UI), it is necessary to understand its effect under different circumstances. In this paper, we focus on a UI based on augmented reality (AR), which can highlight potential hazards on the road. To understand the effect of highlighting on drivers’ SA for objects with different types and locations under various traffic densities, we conducted an in-person experiment with 20 participants on a driving simulator. Our study results show that the effects of highlighting on drivers’ SA varied by traffic densities, object locations and object types. We believe our study can provide guidance in selecting which object to highlight for the AR-based driver-assistance interface to optimize SA for drivers driving and monitoring partially autonomous vehicles.

A lack of sufficient situational awareness is a primary cause of traffic crashes due to human error. Redirecting a driver’s attention to critical objects is essential, but alerting driver about all critical objects can lead to distraction. This paper develops and evaluates an adaptive support system that incorporates drivers’ fixations as a proxy for their situational awareness. We implement an experimental system that detects a driver’s gaze on important objects in the traffic scene and adapts a cueing strategy in an augmented reality-based driver awareness assistance interface. We collect and analyze data from 15 participants and show that our adaptive support system strategy is effective without increasing the drivers’ cognitive workload. Finally, we show that our system can increase ratio of drivers’ fixations on critical objects in their view without significantly increasing dwell time per object.

Situation awareness (SA) is generally considered as the perception, understanding, and projection of objects’ properties and positions. We believe if the system can sense drivers’ SA, it can appropriately provide warnings for objects that drivers are not aware of. To investigate drivers’ awareness, in this study, a human-subject experiment of driving simulation was conducted for data collection. While a previous predictive model for drivers’ situation awareness utilized drivers’ gaze movement only, this work utilizes object properties, characteristics of human visual sensory and memory mechanism. As a result, the proposed driver SA prediction model achieves over 70% accuracy and outperforms the baselines.

Properly calibrated human trust is essential for successful interaction between humans and automation. However, while human trust calibration can be improved by increased automation transparency, too much transparency can overwhelm human workload. To address this tradeoff, we present a probabilistic framework using a partially observable Markov decision process (POMDP) for modeling the coupled trust-workload dynamics of human behavior in an action-automation context. We specifically consider hands-off Level 2 driving automation in a city environment involving multiple intersections where the human chooses whether or not to rely on the automation. We consider automation reliability, automation transparency, and scene complexity, along with human reliance and eye-gaze behavior, to model the dynamics of human trust and workload. We demonstrate that our model framework can appropriately vary automation transparency based on real-time human trust and workload belief estimates to achieve trust calibration.

We developed an experiment to elicit human trust dynamics in human-machine interaction contexts and established a quantitative model of human trust behavior with respect to these contexts. The proposed model describes human trust level as a function of experience, cumulative trust, and expectation bias. We estimated the model parameters using human subject data collected from two experiments. Experiment 1 was designed to excite human trust dynamics using multiple transitions in trust level. Five hundred and eighty-one individuals participated in this experiment. Experiment 2 was an augmentation of Experiment 1 designed to study and incorporate the effects of misses and false alarms in the general model. Three hundred and thirty-three individuals participated in Experiment 2. Beyond considering the dynamics of human trust in automation, this model also characterizes the effects of demographic factors on human trust. In particular, our results show that the effects of national culture and gender on trust are significant. For example, U.S. participants showed a lower trust level and were more sensitive to misses as compared with Indian participants. The resulting trust model is intended for the development of autonomous systems that can respond to changes in human trust level in real time.

Today, intelligent machines interact and collaborate with humans in a way that demands a greater level of trust between human and machine. A first step toward building intelligent machines that are capable of building and maintaining trust with humans is the design of a sensor that will enable machines to estimate human trust level in real time. In this article, two approaches for developing classifier-based empirical trust-sensor models are presented that specifically use electroencephalography and galvanic skin response measurements. Human subject data collected from 45 participants is used for feature extraction, feature selection, classifier training, and model validation. The first approach considers a general set of psychophysiological features across all participants as the input variables and trains a classifier-based model for each participant, resulting in a trust-sensor model based on the general feature set (i.e., a “general trust-sensor model”). The second approach considers a customized feature set for each individual and trains a classifier-based model using that feature set, resulting in improved mean accuracy but at the expense of an increase in training time. This work represents the first use of real-time psychophysiological measurements for the development of a human trust sensor. Implications of the work, in the context of trust management algorithm design for intelligent machines, are also discussed.