Providing the same level of information through the in-vehicle interface can overwhelm automated vehicle occupants in simple scenarios or leave them underinformed in more demanding situations. This study investigates how human preference for in-vehicle feedback detail scales with scene complexity during pedestrian encounters. We measure scene complexity through driving decision diversity and validate its positive correlation with pedestrian crossing intent uncertainty in an initial experiment (N=68). Using a mock-up in-vehicle interface, the second experiment (N=88) evaluates user preferences for manually crafted feedback concepts simulating three levels of the system’s situational awareness. Results indicate that as intent uncertainty increases, users prefer more detailed feedback. While perception-only feedback suffices for simple encounters, in complex situations, information on system comprehension and projection aids better and easier understanding of driving decisions. These findings provide an empirical basis for scaling feedback to situational needs. As this study used manually generated feedback based on ground-truth data, the findings require further investigation considering real-world AI performance in automated vehicles.

Recent advancements in autonomous driving (AD) have explored the use of vision-language models (VLMs) within visual question answering (VQA) frameworks for direct driving decision-making. However, these approaches often depend on handcrafted prompts and suffer from inconsistent performance, limiting their robustness and generalization in real-world scenarios. In this work, we evaluate state-of-the-art open-source VLMs on high-level decision-making tasks using ego-view visual inputs and identify critical limitations in their ability to deliver reliable, context-aware decisions. Motivated by these observations, we propose a new approach that shifts the role of VLMs from direct decision generators to semantic enhancers. Specifically, we leverage their strong general scene understanding to enrich existing vision-based benchmarks with structured, linguistically rich scene descriptions. Building on this enriched representation, we introduce a multi-modal interactive architecture that fuses visual and linguistic features for more accurate decision-making and interpretable textual explanations. Furthermore, we design a post-hoc refinement module that utilizes VLMs to enhance prediction reliability. Extensive experiments on two autonomous driving benchmarks demonstrate that our approach achieves state-of-the-art performance, offering a promising direction for integrating VLMs into reliable and interpretable AD systems.

Accurately modeling pedestrian intention and understanding driver decision-making processes are critical for the development of safe and socially aware autonomous driving systems. However, existing datasets primarily emphasize observable behavior, offering limited insight into the underlying causal reasoning that informs human interpretation and response during traffic interactions. To address this gap, we introduce PSI, a benchmark dataset that captures the dynamic evolution of pedestrian crossing intentions from the driver’s perspective, enriched with human-annotated textual explanations that reflect the reasoning behind intention estimation and driving decision making. These annotations offer a unique foundation for developing and benchmarking models that combine predictive performance with interpretable and human-aligned reasoning. PSI supports standardized tasks and evaluation protocols across multiple dimensions, including pedestrian intention prediction, driver decision modeling, reasoning generation, and trajectory forecasting and more. By enabling causal and interpretable evaluation, PSI advances research toward autonomous systems that can reason, act, and explain in alignment with human cognitive processes.

Rich training datasets are essential for machine learning models to work well. Safety-critical applications, such as autonomous driving, that heavily rely on supervised learning algorithms, require accurate prediction of human behavioral intentions. Training such prediction algorithms requires not only micro-level action labels but also the estimation of human cognitive states and corresponding reasoning explanations at the same granularity level. However, traditional data labeling tools have limited support for these annotation tasks and deficient data structure. This article proposes a novel annotation tool, MindReaD, that facilitates labeling micro-level interactions with free-form reasoning text and grounding text with their visual representations. Focusing on pedestrian–vehicle interactions as a case study, we defined a data structure to label interaction events, collected data from 83 participants using multiple UI designs, and used the data to train ML models for predicting pedestrian crossing intent. The results show that the unique UI design features of Point-and-Explain and multiple prompts can significantly increase the quality of reasoning data annotations. The model trained with data using the proposed data annotation design outperforms the baseline data, with approximately 7% improvement in overall accuracy. User experience study also confirms positive user attitudes towards the developed data annotation system. The data annotation tool will be released to the public.

Electric scooters (e-scooters) have become increasingly popular for intermodal transportation across major US cities, raising safety concerns for both motorists and non-motorists. To develop E-scooter AEB (Autonomous Emergency Braking) systems for intelligent vehicles, it is important to understand the association between e-scooter crash characteristics including facility type, crash severity, and motorist/non-motorist maneuvers. This paper investigates e-scooter crashes to map variable associations. The findings reveal that less severe crashes may be highly underreported. The most critical facilities for e-scooter safety are intersection crosswalks and travel lanes, with the former resulting in more severe injuries. Observing crash trajectories, right-turn crashes are found to be most common at intersections, while only a few intersection crashes result from left-turning vehicles. The findings highlight the need for dedicated safety design and policies for e-scooters, especially for vehicles at intersection crosswalks. Since data on e-scooter crashes is limited, future studies should focus on gathering larger samples from a wider geographic area to investigate and quantify predictor-crash relationships and develop diagnostic and predictive models using statistics and data-driven approaches.

With the development of automated driving systems and V2I (vehicle-to-infrastructure) communications, soft-safety driver alerts can be implemented to supplement imminent driver alerts. This type of alert improves drivers’ situational awareness of emerging risks over a more extended period with more detailed incident information and longer response time. However, compared to the large number of studies focusing on imminent risks, there are insufficient studies on the evaluation and effectiveness of soft-safety alerts. This study proposed an innovative metric to assess the comfort and safety of soft-safety driver alerts by constructing an ideal speed profile and calculating the deviation between the actual and ideal profile. We select the highway end-of-queue event as the experiment scenario, which is a leading cause of fatal highway crashes. Human subjects’ experiments are conducted in the driving simulator to validate the proposed metrics. The results have proved that the proposed metrics have good potential to assess driving comfort objectively. We also found that soft-safety alerts tend to improve driving comfort. However, there is insufficient evidence to conclude statistically about the prototype soft-safety alerts implemented in the experiments.

Pedestrian trajectory prediction is a critical component of autonomous driving in urban environments, allowing vehicles to anticipate pedestrian movements and facilitate safer interactions. While egocentric-view-based algorithms can reduce the sensing and computation burdens of 3D scene reconstruction, accurately predicting pedestrian trajectories and interpreting their intentions from this perspective requires a better understanding of the coupled vehicle (camera) and pedestrian motions, which has not been adequately addressed by existing models. In this paper, we present a novel egocentric pedestrian trajectory prediction approach that uses a two-tower structure and multi-modal inputs. One tower, the vehicle module, receives only the initial pedestrian position and ego-vehicle actions and speed, while the other, the pedestrian module, receives additional prior pedestrian trajectory and visual features. Our proposed action-aware loss function allows the two-tower model to decompose pedestrian trajectory predictions into two parts, caused by ego-vehicle movement and pedestrian movement, respectively, even when only trained on combined ego-view motions. This decomposition increases model flexibility and provides a better estimation of pedestrian actions and intentions, enhancing overall performance. Experiments on three publicly available benchmark datasets show that our proposed model outperforms all existing algorithms in ego-view pedestrian trajectory prediction accuracy.

Protecting vulnerable road users (VRUs) has always been a top priority in intelligent transportation research. Safety-oriented pre-collision systems have been developed and implemented for vehicles with lower automation levels to protect vulnerable road users via kinematics-based crash calculations. Many research efforts, including naturalistic driving study and standardized system evaluation, have been made to ensure the performance of such safety functionalities. Although the warnings and automatic emergency braking systems can efficiently reduce crash injuries, they cannot support higher-level autonomous vehicles interacting with VRUs and driving efficiently without human interventions in urban settings. Numerous algorithms predicting pedestrian trajectories and future actions have been developed in the past several years, which still cannot fully solve the VRU interaction dilemma. In this presentation, we discuss the importance of social intelligence for autonomous vehicles, emphasizing the capability to estimate and negotiate intentions with VRUs for effective and safe interactions. We will highlight the differences between intention and existing behavior predictions, especially regarding autonomous vehicles’ motion planning during vehicle-VRU interactions. A series of research efforts conducted in the Transportation & Autonomous Systems Institute (TASI) at IUPUI will be presented related to developing intention prediction algorithms and benchmark datasets for all types of VRUs. We will propose and discuss open challenges for developing autonomous vehicles with social intelligence.

With rapid development in hardware (sensors and processors) and AI algorithms, automated driving techniques have entered the public’s daily life and achieved great success in supporting human driving performance. However, due to the high contextual variations and temporal dynamics in pedestrian behaviors, the interaction between autonomous-driving cars and pedestrians remains challenging, impeding the development of fully autonomous driving systems. This paper focuses on predicting pedestrian intention with a novel transformer-based evidential prediction (TrEP) algorithm. We develop a transformer module towards the temporal correlations among the input features within pedestrian video sequences and a deep evidential learning model to capture the AI uncertainty under scene complexities. Experimental results on three popular pedestrian intent benchmarks have verified the effectiveness of our proposed model over the state-of-the-art. The algorithm performance can be further boosted by controlling the uncertainty level. We systematically compare human disagreements with AI uncertainty to further evaluate AI performance in confusing scenes. The code is released at https://github.com/zzmonlyyou/TrEP.git.

In recent years, with the help of Ultra-wideband (UWB) radar, contactless measurement of vital signs become possible. This type of radar has sufficient range accuracy for micro-motion such as chest displacement or belly displacement. However, ambient clutter is strong in some scenarios such as a vehicle cabin so that a vital sign of living subject could be obscured and hard to extract. In addition, for multi-person vital signs detection, the sign of people sitting further away from the radar could become weaker due to the strong signal reflected by people in front. This paper describes an efficient way of a vital sign signal localization and a cross-correlation based method to enhance vital signs. The localization method proposed a threshold-based searching method to reduce searching range and save computational power for localization computation. The purpose of vital signs enhancement is 1) to make the weak vital sign signal observable and 2) to increase the signal-to-noise ratio (SNR) of a living subject sitting in at some distance for better localization and vital sign signal extraction accuracy. The enhancement algorithm is to cross-correlate the adjacent bins of the original signal for vital sign signal amplification. Tests are made with multiple people sitting in indoor space with strong reflected material around to simulate vehicle cabin environment. This method has been proved effective for vital sign enhancement in which the SNR increased by more than 40%. In addition, the speed of localization has increased by more than 20% for the two executed test scenarios.

With the significant advances in computer vision research, skeleton model based human pose recognition has become more accurate and time-efficient, although most of the applications are limited in laboratory environment or on surveillance videos. This paper proposes a pose tracking and behavior recognition method from in-vehicle scene camera. It will not only detect pedestrians on the road, but also generate their skeleton models describing head, limb, and trunk movements. Based on these more detailed movements of body parts, the proposed method is designed to track poses of pedestrians and cyclists with the potentials to enable automated pedestrian gesture reading and non-verbal interactions between autonomous vehicles and pedestrians. The proposed algorithm has been tested on different databases including TASI 110-car naturalistic driving database and Joint Attention for Autonomous Driving (JAAD) database. Results show that key frames describing different pedestrian and cyclist negotiation gestures are detected from the raw video streams using the proposed method. These results will improve our understanding of pedestrian and cyclist's intentions and can be further used for autonomous vehicle control algorithm development.

Modeling vehicle-pedestrian interactions in the road environment is essential to develop pedestrian detection and pedestrian crash avoidance systems. In this paper, one novel approach is proposed to estimate the vehicle-pedestrian encountering risk in the road environment based on a large scale naturalistic driving data collection. Considering the difficulty to record actual pedestrian crashes in the naturalistic data collection, the encountering risk is estimated by the chances for driver to meet with pedestrian in the roadway as well as the chances for the driver and pedestrian to get into a potential conflict. Effects of different scenarios consisting of road conditions, pedestrian behaviors, and pedestrian numbers on the risk levels are also evaluated, and significant results are provided.

Integrated multimodal systems is one promising direction to improve human-vehicle interaction. In order to create intelligent human-vehicle interfaces and reduce visual load during secondary tasks, combining a haptic rotary device and a graphic display will provide one practical solution. However, in literature, the proper display position for the haptic rotary device is not fully investigated. In this paper, one experimental infotainment system is studied (including a haptic rotary control device and a graphic display) to evaluate the proper display position. Measurements used include task completion time, reaction to road events, lane/velocity keeping during secondary tasks, and user preference. Three display positions are considered: high mounted position, cluster position, and center stack position. The results show that, with increased on-road and off-road visual loads, the cluster display position can reduce lane position deviation significantly compared to high mounted and center stack positions. In addition, the high mounted and cluster display positions are better toward two different road events, including strong wind gust and extreme deceleration of the lead car.

Investigation of pedestrian step frequency is essential for analyzing walking gaits and pedestrian behaviors. However, most research about step frequency is performed in labs or manually controlled experimental environment, which greatly limits the utilization of the results to analyze and/or predict real pedestrian behaviors. This study investigates the step frequencies of pedestrian in naturalistic driving environment. The mean step frequency values and distribution are studied in all cases and separately for road crossing cases only. Furthermore, comparisons of pedestrian step frequencies are made considering three different impact factors. The results have shown that in real world, people tend to use higher step frequencies when crossing the road, especially when the vehicle is moving towards the pedestrian or when the pedestrians are crossing without right-of-way.

Driver distraction detection and intervention are important for designing modern driver-assistance systems and for improving safety. The main research question of this paper is to investigate how the cumulative driver off-road glance duration can be controlled to reduce the probability of occurrences of crash and near-crash events. Based on the available data sets from the Virginia Tech Transportation Institute (VTTI) 100-car study, the conditional probability is calculated to study the chance of crash and near-crash events when the given cumulative off-road glance duration in 6 s has been reached. Different off-road eye-glance locations and traffic density levels are also evaluated. The results show that one linear relationship can be obtained between the cumulative off-road eye-glance duration in 6 s and the risk of occurrences of crash and near-crash events, which varies for different off-road eye-glance locations. In addition, the traffic density level is found to be one significant moderator to this linear relationship. Detailed comparisons are made for different traffic density levels, and one nonlinear equation is obtained to predict the probability of occurrences of crash and near-crash events by considering both cumulative off-road glance duration and traffic density levels.