As autonomous vehicles (AVs) advance, the integration of Large (Vision) Language Models (LLMs and VLMs) has emerged as a promising approach to enhance AV capabilities in perception, planning, decision-making, and data generation. However, the practical challenges of incorporating LLMs and VLMs into AV systems, including computational efficiency, real-time processing, and ethical considerations, remain underexplored. This survey aims to provide a comprehensive review of the current research on LLM and VLM applications in AVs, focusing on the following key areas: modular integration, end-to-end integration, data generation, evaluation platforms, datasets, and benchmarks. We systematically analyse 77 recent papers published before Sep 2025, detailing their methodologies and models. Our findings highlight the potential of LLMs and VLMs to improve AV system performance while acknowledging limitations. This survey offers researchers and practitioners a panoptic view of the classification and progression of LLMs and VLMs in the AV sphere, while systematically distilling models to their core components. We envision this survey as a central reference for AV researchers navigating this rapidly evolving landscape to accelerate future research.

Knowledge of lane topology is a core problem in au-tonomous driving. Aerial imagery can provide high res-olution, quickly updatable lane source data but detecting lanes from such data has so far been an expensive man-ual process or, where automated solutions exist, undriv-able and requiring of downstream processing. We pro-pose a method for large-scale lane topology extraction from aerial imagery while ensuring that the resulting lanes are realistic and drivable by introducing a novel Bezier Graph shared parameterisation of Bezier curves. We develop a transformer-based model to predict these Bezier Graphs from input aerial images, demonstrating competitive results on the UrbanLaneGraph dataset. We demonstrate that our method generates realistic lane graphs which require both minimal input, and minimal downstream processing. We make our code publicly available at https://github.com/driskai/BGFormer

The efficiency of urban traffic management and congestion alleviation relies heavily on accurate forecasting of Origin-Destination (O-D) demand matrices. Existing models primarily focus on estimating O-D demand for various travel purposes throughout the day, which is characterised by its pulsating nature. However, these models often compromise the precision of peak-hour forecasts, leading to unreliable dynamic traffic control and challenges in effectively reducing peak-hour congestion. To tackle this challenge, this paper proposes a novel method for predicting commuting O-D demand matrices. Our method employs community detection algorithms on road networks to precisely partition commute O-D regions, incorporating Points of Interest (POIs). We also present a spatio-temporal dynamic weighted hypergraph model that leverages these partitioned regions, time characteristics from observed O-D trips, and meteorological data to improve forecasting. Comparative analyses with contemporary models and ablation studies indicate our method significantly enhances prediction accuracy, by approximately 5%. These findings imply that the proposed method more effectively encompasses the varied characteristics of commuting during peak hours, thereby providing more accurate demand matrices for urban traffic management.

Autonomous Vehicles (AVs) have the potential to save millions of lives and increase the efficiency of transportation services. However, the successful deployment of AVs requires tackling multiple challenges related to modeling and certifying safety. State-of-the-art decision-making methods usually rely on end-to-end learning or imitation learning approaches, which still pose significant safety risks. Hence the necessity of risk-aware AVs that can better predict and handle dangerous situations. Furthermore, current approaches tend to lack explainability due to their reliance on end-to-end Deep Learning, where significant causal relationships are not guaranteed to be learned from data. This paper introduces a novel risk-aware framework for training AV agents using a bespoke collision prediction model and Reinforcement Learning (RL). The collision prediction model is based on Gaussian Processes and vehicle dynamics, and is used to generate the RL state vector. Using an explicit risk model increases the post-hoc explainability of the AV agent, which is vital for reaching and certifying the high safety levels required for AVs and other safety-sensitive applications. Experimental results obtained with a simulator and state-of-the-art RL algorithms show that the risk-aware RL framework decreases average collision rates by 15%, makes AVs more robust to sudden harsh braking situations, and achieves better performance in both safety and speed when compared to a standard rule-based method (the Intelligent Driver Model). Moreover, the proposed collision prediction model outperforms other models in the literature.

Pedestrian trajectory prediction is an important area in computer vision, with wide applications in autonomous driving, robot path planning, and surveillance systems. The core underlying technique of these applications is pattern recognition. A key challenge in this area is modeling social interactions between pedestrians, such as pedestrian view area and group behaviors. However, although many methods have been proposed to model social interactions, pedestrian view area and group behaviors have not been explored together to account for complex situations. Additionally, most existing studies require additional detectors and manual annotations to handle view area and group interactions, respectively. In this paper, we propose a dual-branch spatio-temporal graph neural network to automatically model view area and grouping together. Specifically, a spatio-temporal graph attention network (STGAT) branch is designed to handle pedestrian view area, and a spatio-temporal graph convolutional network (STGCN) branch is designed to model group interactions. The features of these branches are then fused to provide better feature representations, on which a temporal convolution operation (TCN) is performed for trajectory prediction. Experiments on public standard datasets demonstrate that the proposed method achieves very competitive performance and predicts socially acceptable trajectories in different challenging scenarios.

The deployment of Autonomous Vehicles (AVs) poses considerable challenges and unique opportunities for the design and management of future urban road infrastructure. In light of this disruptive transformation, the Right-Of-Way (ROW) composition of road space has the potential to be renewed. Design approaches and intelligent control models have been proposed to address this problem, but we lack an operational framework that can dynamically generate ROW plans for AVs and pedestrians in response to real-time demand. Based on microscopic traffic simulation, this study explores Reinforcement Learning (RL) methods for evolving ROW compositions. We implement a centralised learning paradigm and a distributive learning paradigm to separately perform the dynamic control on several road network configurations. Experimental results indicate that the algorithms have the potential to improve traffic flow efficiency and allocate more space for pedestrians. Furthermore, the distributive learning algorithm outperforms its centralised counterpart regarding computational cost (49.55%), benchmark rewards (25.35%), best cumulative rewards (24.58%), optimal actions (13.49%) and rate of convergence. This novel road management technique could potentially contribute to the flow-adaptive and active mobility-friendly streets in the AVs era.

Autonomous Driving requires high levels of coordination and collaboration between agents. Achieving effective coordination in multi-agent systems is a difficult task that remains largely unresolved. Multi-Agent Reinforcement Learning has arisen as a powerful method to accomplish this task because it considers the interaction between agents and also allows for decentralized training—which makes it highly scalable. However, transferring policies from simulation to the real world is a big challenge, even for single-agent applications. Multi-agent systems add additional complexities to the Sim-to-Real gap due to agent collaboration and environment synchronization. In this paper, we propose a method to transfer multi-agent autonomous driving policies to the real world. For this, we create a multi-agent environment that imitates the dynamics of the Duckietown multi-robot testbed, and train multi-agent policies using the MAPPO algorithm with different levels of domain randomization. We then transfer the trained policies to the Duckietown testbed and show that when using our method, domain randomization can reduce the reality gap by 90%. Moreover, we show that different levels of parameter randomization have a substantial impact on the Sim-to-Real gap. Finally, our approach achieves significantly better results than a rule-based benchmark.

Ride-sourcing platforms often face imbalances in the demand and supply of rides across areas in their operating road-networks. As such, dynamic pricing methods have been used to mediate these demand asymmetries through surge price multipliers, thus incentivising higher driver participation in the market. However, the anticipated commercialisation of autonomous vehicles could transform the current ride-sourcing platforms to fleet operators. The absence of human drivers fosters the need for empty vehicle management to address any vehicle supply deficiencies. Proactive redistribution using integer programming and demand predictive models have been proposed in research to address this problem. A shortcoming of existing models, however, is that they ignore the market structure and underlying customer choice behaviour. As such, current models do not capture the real value of redistribution. To resolve this, we formulate the vehicle redistribution problem as a non-linear minimum cost flow problem which accounts for the relationship of supply and demand of rides, by assuming a customer discrete choice model and a market structure. We demonstrate that this model can have a convex domain, and we introduce an edge splitting algorithm to solve a transformed convex minimum cost flow problem for vehicle redistribution. By testing our model using simulation, we show that our redistribution algorithm can decrease wait times by more than 50%, increase profit up to 10% with less than 20% increase in vehicle mileage. Our findings outline that the value of redistribution is contingent on localised market structure and customer behaviour.

Pedestrian crossing intention prediction is crucial for the safety of pedestrians in the context of both autonomous and conventional vehicles and has attracted widespread interest recently. Various methods have been proposed to perform pedestrian crossing intention prediction, among which the skeleton-based methods have been very popular in recent years. However, most existing studies utilize manually designed features to handle skeleton data, limiting the performance of these methods. To solve this issue, we propose to predict pedestrian crossing intention based on spatial-temporal graph convolutional networks using skeleton data (ST CrossingPose). The proposed method can learn both spatial and temporal patterns from skeleton data, thus having a good feature representation ability. Extensive experiments on a public dataset demonstrate that the proposed method achieves very competitive performance in predicting crossing intention while maintaining a fast inference speed. We also analyze the effect of several factors, e.g., size of pedestrians, time to event, and occlusion, on the proposed method.

To facilitate the monitoring and management of modern transportation systems, monocular visual traffic surveillance systems have been widely adopted for speed measurement, accident detection, and accident prediction. Thanks to the recent innovations in computer vision and deep learning research, the performance of visual traffic surveillance systems has been significantly improved. However, despite this success, there is a lack of survey papers that systematically review these new methods. Therefore, we conduct a systematic review of relevant studies to fill this gap and provide guidance to future studies. This paper is structured along the visual information processing pipeline that includes object detection, object tracking, and camera calibration. Moreover, we also include important applications of visual traffic surveillance systems, such as speed measurement, behavior learning, accident detection and prediction. Finally, future research directions of visual traffic surveillance systems are outlined.

Transportation Network Companies employ dynamic pricing methods at periods of peak travel to incentivise driver participation and balance supply and demand for rides. Surge pricing multipliers are commonly used and are applied following demand and estimates of customer and driver trip valuations. Combinatorial double auctions have been identified as a suitable alternative, as they can achieve maximum social welfare in the allocation by relying on customers and drivers stating their valuations. A shortcoming of current models, however, is that they fail to account for the effects of trip detours that take place in shared trips and their impact on the accuracy of pricing estimates. To resolve this, we formulate a new shared-ride assignment and pricing algorithm using combinatorial double auctions. We demonstrate that this model is reduced to a maximum weighted independent set model, which is known to be APX-hard. A fast local search heuristic is also presented, which is capable of producing results that lie within 10% of the exact approach for practical implementations. Our proposed algorithm could be used as a fast and reliable assignment and pricing mechanism of ride-sharing requests to vehicles during peak travel times.