In this work, we present WeatherEdit, a novel weather editing pipeline for generating realistic weather effects with controllable types and severity in 3D scenes. Our approach is structured into two key components: weather background editing and weather particle construction. For weather background editing, we introduce an all-in-one adapter that integrates multiple weather styles into a single diffusion model, enabling the generation of diverse weather effects in 2D image backgrounds. During inference, we design a Temporal-View (TV-) attention mechanism that follows a specific order to aggregate temporal and spatial information, ensuring consistent editing across multi-frame and multi-view images. To construct the weather particles, we first reconstruct a 3D scene using the edited images and then introduce a 4D Gaussian field to generate snowflakes, raindrops and fog in the scene. The attributes and dynamics of these particles are controlled through attribute modelling and dynamic simulation, ensuring realistic weather representation and flexible severity adjustments. Finally, we integrate the 4D Gaussian field with the 3D scene to render consistent and highly realistic weather effects. Experiments on multiple driving datasets demonstrate that WeatherEdit can generate diverse weather effects with controllable condition severity, highlighting its potential for autonomous driving simulation in adverse weather.

D Gaussian Splatting (3DGS) has gained significant attention for 3D scene reconstruction, but still suffers from complex outdoor environments, especially under adverse weather. This is because 3DGS treats the artifacts caused by adverse weather as part of the scene and will directly reconstruct them, largely reducing the clarity of the reconstructed scene. To address this challenge, we propose WeatherGS, a 3DGSbased framework for reconstructing clear scenes from multiview images under different weather conditions. Specifically, we explicitly categorize the multi-weather artifacts into the dense particles and lens occlusions that have very different characters, in which the former are caused by snowflakes and raindrops in the air, and the latter are raised by the precipitation on the camera lens. In light of this, we propose a dense-to-sparse preprocess strategy, which sequentially removes the dense particles by an Atmospheric Effect Filter (AEF) and then extracts the relatively sparse occlusion masks with a Lens Effect Detector (LED). Finally, we train a set of 3D Gaussians by the processed images and generated masks for excluding occluded areas, and accurately recover the underlying clear scene by Gaussian splatting. We conduct a diverse and challenging benchmark to facilitate the evaluation of 3D reconstruction under complex weather scenarios. Extensive experiments on this benchmark demonstrate that our WeatherGS consistently produces high-quality, clean scenes across various weather scenarios, outperforming existing state-of-the-art methods. See project: https://jumponthemoon.github.io/weather-gs.

Understanding driver–pedestrian interactions at unsignalized locations has gained additional importance due to recent advancements in vehicle automation. Naturalistic observations can only provide correlational data of limited value for understanding and modeling the mechanisms underlying road user interaction. Therefore, controlled studies in virtual reality (VR) are an important complement, but conventional methods can only accommodate a single human participant. Recently, there has been some interest in studying interactions in VR, by means of distributed simulation, involving multiple human participants. However, there is a lack of validation of this method. Here, we provide a validation study, focusing on a distributed vehicle–pedestrian interaction setup, where pairs of one driver and one pedestrian interacted under various kinematic conditions in a connected virtual environment. To test the validity of the distributed simulation, we used a naturalistic dataset collected in the same U.K. city, at similar locations, and compared the observed behavior between the two settings. Our results indicate a good relative validity of the simulator study, where road users showed similar nonverbal communication behavior in both datasets. As an additional means of validation, we also leveraged a set of game theoretic models that were developed based on the simulator studies, and found that when applied to the naturalistic dataset, we obtained similar (although not identical) model selection results. The findings suggest that distributed simulation can also be useful for development of computational models of interaction. Overall, the findings suggest that distributed simulation can be a highly valuable tool for studying and modeling road user interactions.

As autonomous driving technology advances, automated vehicles (AVs) will increasingly share road space with pedestrians, creating significant challenges for AV systems. Effective interaction between AVs and pedestrians is one of the key hurdles. Pedestrian simulation tools offer the potential to expedite the evaluation and refinement of these interactive capabilities. However, existing research lacks efforts to model pedestrian behavior in vehicle-yielding scenarios, resulting in distorted modeling results. This paper proposes a perceptually plausible road-crossing decision model that creates temporal-dynamic crossing decisions across a range of vehicle-yielding scenarios. Specifically, a proposed hybrid perception strategy explains how pedestrians may apply psychophysical cues to make crossing decisions. Discrete choice models based on the hybrid perception strategy combined with a crossing initiation model reproduce the details of crossing decisions: the decision and its timing. An empirical dataset collected in a pedestrian simulator is applied to validate the model. Additionally, the latest crossing decision models, i.e., the evidence accumulation model and the artificial neural networks approach, are employed as comparisons. The results show that the proposed model accurately reproduces crossing decision patterns affected by diverse vehicle kinematics in vehicle-yielding scenarios in a perceptually plausible manner. Our results strengthen the notion that there is a perceptual threshold for pedestrians to control their decision-making strategy. The proposed theory and approach bring insights into the computational pedestrian road-crossing behavior and have practical implications in traffic simulation and AV development.

Understanding pedestrian behavior is crucial for the safe deployment of Autonomous Vehicles (AVs) in urban environments. Traditional pedestrian behavior models often fall into two categories: mechanistic models, which do not generalize well to complex environments, and machine-learned models, which generally overlook sensory-motor constraints influencing human behavior and which are thus prone to fail in unseen scenarios. We hypothesize that sensory-motor constraints, fundamental to how humans perceive and interact with their surroundings, are essential for realistic simulations. Thus, we introduce a constrained reinforcement learning (RL) model that simulates the crossing decision and locomotion of pedestrians. Our model includes human sensory constraints, giving the agent imperfect information about the environment, and human motor constraints incorporated through a bio-mechanical model of walking. We gathered data from a human-in-the-loop experiment to understand pedestrian behavior. The findings reveal several behavioral patterns not addressed by existing pedestrian models, regarding how pedestrians adapt their walking speed to the kinematics and behavior of the approaching vehicle. Our model successfully captures these human-like walking speed patterns, enabling us to understand these patterns as a trade-off between time pressure and walking effort. Importantly, the model with both sensory and motor constraints performed better than models only incorporating one of the two. Additionally, behavioral patterns related to external human-machine interfaces and light conditions were also captured by the model. Overall, our results not only demonstrate the potential of constrained RL in modeling pedestrian behaviors but also highlight the importance of sensory-motor mechanisms in modeling pedestrian-vehicle interactions.

In this paper we introduce a general estimation methodology for learning a model of human perception and control in a sensorimotor control task based upon a finite set of demonstrations. The model’s structure consists of (i) the agent’s internal representation of how the environment and associated observations evolve as a result of control actions and (ii) the agent’s preferences over observable outcomes. We consider a model’s structure specification consistent with active inference, a theory of human perception and behavior from cognitive science. According to active inference, the agent acts upon the world so as to minimize surprise defined as a measure of the extent to which an agent’s current sensory observations differ from its preferred sensory observations. We propose a bi-level optimization approach to estimation which relies on a structural assumption on prior distributions that parameterize the statistical accuracy of the human agent’s model of the environment. To illustrate the proposed methodology, we present the estimation of a model for car-following behavior based upon a naturalistic dataset. Overall, the results indicate that learning active inference models of human perception and control from data is a promising alternative to closed-box models of driving.

The decision of whether to cross a road or wait for a car to pass, humans make frequently and effortlessly. Recently, the application of drift-diffusion models (DDMs) on pedestrians’ decision-making has proven useful in modelling crossing behaviour in pedestrian-vehicle interactions. These models consider binary decision-making as an incremental accumulation of noisy evidence over time until one of two choice thresholds (to cross or not) is reached. One open question is whether the assumption of a kinematics-dependent drift-diffusion process, which was made in previous pedestrian crossing DDMs, is justified, with DDM-parameters varying over time according to the developing traffic situation. It is currently unknown whether kinematics-dependent DDMs provide a better model fit than conventional DDMs, which are fitted per condition. Furthermore, previous DDMs have not considered reaction times for the not-crossing option. We address these issues by a novel experimental design combined with modelling. Experimentally, we use a 2-alternative-forced-choice paradigm, where participants view videos of approaching cars from a pedestrian’s perspective and respond whether they want to cross before the car or to wait until the car has passed. Using these data, we perform thorough model comparison between kinematics-dependent and condition-wise fitted DDMs. Our results demonstrate that condition-wise fitted DDMs can show better model fits than kinematics-dependent DDMs as reflected in the mean-squared-errors. The condition-wise fitted models need considerably more parameters, but in some cases still outperform kinematics-dependent DDMs in measures that penalize the parameter number (e.g., Akaike information criterion). Introducing a starting point bias provides support for the novel hypothesis of rapid early evidence build-up from the initial view of the vehicle distance. The drift rates obtained for the condition-wise fitted models align with the assumptions in the kinematics-dependent models, confirming that pedestrians’ decision processes are kinematics-dependent. However, the partial preference for condition-wise fitted models in the model selection suggests that the correct form of kinematics-dependence has not yet been identified for all DDM-parameters, indicating room for improvement of current pedestrian crossing DDMs. Developing more accurate models of human cognitive processes will likely facilitate autonomous vehicles to understand pedestrians’ intentions as well as to show unambiguous human-like behaviour in future traffic interactions with humans.

Increasing attention has been drawn to computational pedestrian behavior models aimed at understanding the interaction mechanisms between pedestrians and vehicles. Nevertheless, existing research lacks exploration of the underlying behavioral mechanisms of pedestrian crossing decisions, which leads to unrealistic modeling results. In particular, when dealing with continuous traffic flow scenarios, the concept of waiting time is frequently used to account for all intricate traffic flow effects. Moreover, very few studies considered the time-dynamic nature of crossing decisions. To address these research limitations, this study deconstructs pedestrian crossing decisions at uncontrolled intersections with continuous traffic flow through a cognitive process and proposes an anthropomorphic crossing decision model. Specifically, we propose a novel visual collision cue-based crossing decision-initiation model to characterize time-dynamic crossing decisions. In light of the risk-aversion theory, a traffic gap comparison strategy is put forward to explain and model pedestrian waiting behavior in traffic flow. Two datasets collected from a CAVE-based immersive pedestrian simulator are applied to calibrate and validate the model. The proposed model accurately predicts pedestrian crossing decisions across all traffic scenarios. The modeling performance is significantly enhanced by considering the proposed traffic gap comparison strategy. Moreover, the model accurately captures the timing of crossing decisions. This work concisely demonstrates how pedestrians dynamically adapt their crossings in continuous traffic based on visual collision cues, potentially offering insights into modeling pedestrian-vehicle interactions or serving as a tool to realize anthropomorphic pedestrian crossing decisions in simulators.

Understanding the interaction between different road users is critical for road safety and automated vehicles (AVs). Existing mathematical models on this topic have been proposed based mostly on either cognitive or machine learning (ML) approaches. However, current cognitive models are incapable of simulating road user trajectories in general scenarios, and ML models lack a focus on the mechanisms generating the behavior and take a high-level perspective which can cause failures to capture important human-like behaviors. Here, we develop a model of human pedestrian crossing decisions based on computational rationality, an approach using deep reinforcement learning (RL) to learn boundedly optimal behavior policies given human constraints, in our case a model of the limited human visual system. We show that the proposed combined cognitive-RL model captures human-like patterns of gap acceptance and crossing initiation time. Interestingly, our model’s decisions are sensitive to not only the time gap, but also the speed of the approaching vehicle, something which has been described as a “bias” in human gap acceptance behavior. However, our results suggest that this is instead a rational adaption to human perceptual limitations. Moreover, we demonstrate an approach to accounting for individual differences in computational rationality models, by conditioning the RL policy on the parameters of the human constraints. Our results demonstrate the feasibility of generating more human-like road user behavior by combining RL with cognitive models.

Predicting pedestrian behavior when interacting with vehicles is one of the most critical challenges in the field of automated driving. Pedestrian crossing behavior is influenced by various interaction factors, including time to arrival, pedestrian waiting time, the presence of zebra crossing, and the properties and personality traits of both pedestrians and drivers. However, these factors have not been fully explored for use in predicting interaction outcomes. In this paper, we use machine learning to predict pedestrian crossing behavior including pedestrian crossing decision, crossing initiation time (CIT), and crossing duration (CD) when interacting with vehicles at unsignalized crossings. Distributed simulator data are utilized for predicting and analyzing the interaction factors. Compared with the logistic regression baseline model, our proposed neural network model improves the prediction accuracy and F1 score by 4.46% and 3.23%, respectively. Our model also reduces the root mean squared error (RMSE) for CIT and CD by 21.56% and 30.14% compared with the linear regression model. Additionally, we have analyzed the importance of interaction factors, and present the results of models using fewer factors. This provides information for model selection in different scenarios with limited input features.

Autonomous vehicles use a variety of sensors and machine-learned models to predict the behavior of surrounding road users. Most of the machine-learned models in the literature focus on quantitative error metrics like the root mean square error (RMSE) to learn and report their models’ capabilities. This focus on quantitative error metrics tends to ignore the more important behavioral aspect of the models, raising the question of whether these models really predict human-like behavior. Thus, we propose to analyze the output of machine-learned models much like we would analyze human data in conventional behavioral research. We introduce quantitative metrics to demonstrate presence of three different behavioral phenomena in a naturalistic highway driving dataset: 1) The kinematics-dependence of who passes a merging point first 2) Lane change by an on-highway vehicle to accommodate an on-ramp vehicle 3) Lane changes by vehicles on the highway to avoid lead vehicle conflicts. Then, we analyze the behavior of three machine-learned models using the same metrics. Even though the models’ RMSE value differed, all the models captured the kinematic-dependent merging behavior but struggled at varying degrees to capture the more nuanced courtesy lane change and highway lane change behavior. Additionally, the collision aversion analysis during lane changes showed that the models struggled to capture the physical aspect of human driving: leaving adequate gap between the vehicles. Thus, our analysis highlighted the inadequacy of simple quantitative metrics and the need to take a broader behavioral perspective when analyzing machine-learned models of human driving predictions.

Evidence accumulation models provide a dominant account of human decision-making, and have been particularly successful at explaining behavioral and neural data in laboratory paradigms using abstract, stationary stimuli. It has been proposed, but with limited in-depth investigation so far, that similar decision-making mechanisms are involved in tasks of a more embodied nature, such as movement and locomotion, by directly accumulating externally measurable sensory quantities of which the precise, typically continuously time-varying, magnitudes are important for successful behavior. Here, we leverage collision threat detection as a task which is ecologically relevant in this sense, but which can also be rigorously observed and modelled in a laboratory setting. Conventionally, it is assumed that humans are limited in this task by a perceptual threshold on the optical expansion rate-the visual looming-of the obstacle. Using concurrent recordings of EEG and behavioral responses, we disprove this conventional assumption, and instead provide strong evidence that humans detect collision threats by accumulating the continuously time-varying visual looming signal. Generalizing existing accumulator model assumptions from stationary to time-varying sensory evidence, we show that our model accounts for previously unexplained empirical observations and full distributions of detection response. We replicate a pre-response centroparietal positivity (CPP) in scalp potentials, which has previously been found to correlate with accumulated decision evidence. In contrast with these existing findings, we show that our model is capable of predicting the onset of the CPP signature rather than its buildup, suggesting that neural evidence accumulation is implemented differently, possibly in distinct brain regions, in collision detection compared to previously studied paradigms.

Autonomous vehicles (AVs) must share space with pedestrians, both in carriageway cases such as cars at pedestrian crossings and off-carriageway cases such as delivery vehicles navigating through crowds on pedestrianized high-streets. Unlike static obstacles, pedestrians are active agents with complex, interactive motions. Planning AV actions in the presence of pedestrians thus requires modelling of their probable future behavior as well as detecting and tracking them. This narrative review article is Part II of a pair, together surveying the current technology stack involved in this process, organising recent research into a hierarchical taxonomy ranging from low-level image detection to high-level psychological models, from the perspective of an AV designer. This self-contained Part II covers the higher levels of this stack, consisting of models of pedestrian behavior, from prediction of individual pedestrians' likely destinations and paths, to game-theoretic models of interactions between pedestrians and autonomous vehicles. This survey clearly shows that, although there are good models for optimal walking behavior, high-level psychological and social modelling of pedestrian behavior still remains an open research question that requires many conceptual issues to be clarified. Early work has been done on descriptive and qualitative models of behavior, but much work is still needed to translate them into quantitative algorithms for practical AV control.

Efficient and safe interactions between automated vehicles and other road users can be supported through external Human-Machine Interfaces (eHMI). The success of these interactions relies on the eHMI signals being adequately understood by other road users. A paired-comparison forced choice task (Task 1), and a 6-point rating task (Task 2) were used to assess the extent to which ten different eHMI signals conveyed three separate messages, 'I am giving way', 'I am in automated mode' and 'I will start moving'. The different eHMI options consisted of variations of a 360° lightband, a single lamp, and an auditory signal. Results demonstrated that the same eHMI format could convey different messages equally well, suggesting a need to be cautious when designing eHMI, to avoid presenting misleading, potentially unsafe, information. Future research should investigate whether the use of an eHMI signal indicating a change in the AV's behaviour is sufficient for conveying intention.

Human operators often employ intermittent, discontinuous control strategies in a variety of tasks. A typical intermittent controller monitors control error and generates corrective action when the deviation of the controlled system from the desired state becomes too large to ignore. Most contemporary models of human intermittent control employ simple, threshold-based trigger mechanism to model the process of control activation. However, recent experimental studies demonstrate that the control activation patterns produced by human operators do not support threshold-based models, and provide evidence for more complex activation mechanisms. In this paper, we investigate whether intermittent control activation in humans can be modeled as a decision-making process. We utilize an established drift-diffusion model, which treats decision making as an evidence accumulation process, and study it in simple numerical simulations. We demonstrate that this model robustly replicates the control activation patterns (distributions of control error at movement onset) produced by human operators in previously conducted experiments on virtual inverted pendulum balancing. Our results provide support to the hypothesis that intermittent control activation in human operators can be treated as an evidence accumulation process.

Autonomous vehicle control is well understood for local- [15], good approximations exist such as particle ?ltering,ization, mapping and planning in un-reactive environ- which make use of large compute power to draw samplesments, but the human factors of complex interactions near solutions.stood [16], and despite its exact solution being NP-hardwith other road users are not yet developed.Route planning in non-interactive envi-ronments also has well known tractable solutions such asThis po-the A-star algorithm. Given a route, localizing and con-sition paper presents an initial model for negotiation be-trol to follow that route then becomes a similar task totween an autonomous vehicle and another vehicle at anthat performed by the 1959 General Motors Firebird-IIIunsigned intersections or (equivalently) with a pedestrianself-driving car [1], which used electromagnetic sensingat an unsigned road-crossing (jaywalking), using discreteto follow a wire built into the road.Such path follow-sequential game theory. The model is intended as a ba- ing, using wires or SLAM, can then be augmented withsic framework for more realistic and data-driven future simple safety logic to stop the vehicle if any obstacle isextensions. The model shows that when only vehicle po- in its way, as detected by any range sensor.sition is used to signal intent, the optimal behaviors for open source systems for this level of `self-driving' are nowboth agents must include a non-zero probability of al- widely available [6].lowing a collision to occur.In contrast,This suggests extensions toproblems that these vehicles will facearound interacting with other road users are much harderreduce this probability in future, such as other forms ofboth to formulate and solve. Autonomous vehicles do notsignaling and control. Unlike most Game Theory appli-just have to deal with inanimate objects, sensors, andcations in Economics, active vehicle control requires real-maps.time selection from multiple equilibria with no history,They have to deal with other agents, currentlyhuman drivers and pedestrians and eventually other au-and we present and argue for a novel solution concept,meta-strategy convergence , suited to this task.

For a driving simulator to be a valid tool for research, vehicle development, or driver training, it is crucial that it elicits similar driver behavior as the corresponding real vehicle. To assess such behavioral validity, the use of quantitative driver models has been suggested but not previously reported. Here, a task-general conceptual driver model is proposed, along with a taxonomy defining levels of behavioral validity. Based on these theoretical concepts, it is argued that driver models without explicit representations of sensory or neuromuscular dynamics should be sufficient for a model-based assessment of driving simulators in most contexts. As a task-specific example, two parsimonious driver steering models of this nature are developed and tested on a dataset of real and simulated driving in near-limit, low-friction circumstances, indicating a clear preference of one model over the other. By means of closed-loop simulations, it is demonstrated that the parameters of this preferred model can generally be accurately estimated from unperturbed driver steering data, using a simple, open-loop fitting method, as long as the vehicle positioning data are reliable. Some recurring patterns between the two studied tasks are noted in how the model's parameters, fitted to human steering, are affected by the presence or absence of steering torques and motion cues in the simulator.

An experiment was carried out on a low friction test track, where seven truck drivers repeatedly performed collision avoidance and stabilization with a 4×2 tractor. A previous finding from a simulator study was confirmed: In severe yaw instability, drivers engaged in a yaw rate nulling type of steering behavior, in conflict with the assumptions of conventional electronic stability control (ESC), and the experiment provided indications of conventional ESC behaving suboptimally in these situations. Promising results were obtained for modified versions of the ESC, based on the yaw rate nulling model of steering, but further development work is needed.

The Adaptive Integrated Driver-vehicle interfacE (AIDE) is an integrated project funded by the European Commission in the Sixth Framework Programme. The project, which involves 31 partners from the European automotive industry and academia, deals with behavioral and technical issues related to automotive human-machine interface (HMI) design, with a particular focus on integration and adaptation. The project involves tightly integrated empirical research, driver-behavior modeling, and methodological and technological development. This paper provides an overview of the AIDE Sub-Project 3 results dealing with the design, development, and integration of the AIDE system in three prototype vehicles, together with the evaluation results of the trials.

Driver assistance systems and electronics (e.g. navigators, cell phones, etc.) steal increasing amounts of driver attention. Therefore, the vehicle industry is striving to build a driving environment where input-output devices are smartly scheduled, allowing sufficient time for the driver to focus attention on the surrounding traffic. To enable a smart human-machine interface (HMI), the driver's momentary state needs to be measured. This paper describes a facility for monitoring the distraction of a driver and presents some early evaluation results. The module is able to detect the driver's visual and cognitive workload by fusing stereo vision and lane tracking data, running both rule-based and support-vector machine (SVM) classification methods. The module has been tested with data from a truck and a passenger car. The results show over 80% success in detecting visual distraction and a 68-86 % success in detecting cognitive distraction, which are satisfactory results.