With the broader adoption and highly successful development of large language models (LLMs), there has been growing interest and demand for applying LLMs to autonomous driving technology. Driven by their natural language (NL) understanding and reasoning capabilities, LLMs have the potential to enhance various aspects of autonomous driving systems, from perception and scene understanding to interactive decision-making. This article first introduces the novel concept of designing LLMs for autonomous driving (LLM4AD), followed by a review of existing LLM4AD studies. Then, a comprehensive benchmark is proposed for evaluating the instruction-following and reasoning abilities of LLM4AD systems, which includes LaMPilot-Bench, CARLA Leaderboard 1.0 Benchmark in simulation and NuPlanQA for multiview visual question answering (VQA). Furthermore, extensive real-world experiments are conducted on autonomous vehicle platforms, examining both on-cloud and on-edge LLM deployment for personalized decision-making and motion control. Next, the future trends of integrating language diffusion models into autonomous driving are explored, exemplified by the proposed vision-language diffusion (ViLaD) framework. Finally, the main challenges of LLM4AD are discussed, including latency, deployment, security and privacy, safety, trust and transparency, and personalization.

Large Language Models (LLMs) are increasingly integrated into everyday applications, but their prevalent cloud-based deployment raises growing concerns around data privacy and long-term sustainability. Running LLMs locally on mobile and edge devices (on-device LLMs) offers the promise of enhanced privacy, reliability, and reduced communication costs. However, realizing this vision remains challenging due to substantial memory and compute demands, as well as limited visibility into performance-efficiency trade-offs on resource-constrained hardware. We propose lm-Meter, the first lightweight, online latency profiler tailored for on-device LLM inference. lm-Meter captures fine-grained, real-time latency at both phase (e.g., embedding, prefill, decode, softmax, sampling) and kernel levels without auxiliary devices. We implement lm-Meter on commercial mobile platforms and demonstrate its high profiling accuracy with minimal system overhead, e.g., only 2.58% throughput reduction in prefill and 0.99% in decode under the most constrained Powersave governor. Leveraging lm-Meter, we conduct comprehensive empirical studies revealing phase- and kernel-level bottlenecks in on-device LLM inference, quantifying accuracy-efficiency trade-offs, and identifying systematic optimization opportunities. lm-Meter provides unprecedented visibility into the runtime behavior of LLMs on constrained platforms, laying the foundation for informed optimization and accelerating the democratization of on-device LLM systems. Code and tutorials are available at github.com/amai-gsu/LM-Meter.

Hardware-Aware Neural Architecture Search (HW-NAS) has emerged as a powerful tool for designing efficient deep neural networks (DNNs) tailored to edge devices. However, existing methods remain largely impractical for real-world deployment due to their high time cost, extensive manual profiling, and poor scalability across diverse hardware platforms with complex, device-specific energy behavior.

The rapid iteration of autonomous vehicle (AV) deployments leads to increasing needs for building realistic and scalable multi-agent traffic simulators for efficient evaluation. Recent advances in this area focus on closed-loop simulators that enable generating diverse and interactive scenarios. This paper introduces Neural Interactive Agents (NIVA), a probabilistic framework for multi-agent simulation driven by a hierarchical Bayesian model that enables closed-loop, observation-conditioned simulation through autoregressive sampling from a latent, finite mixture of Gaussian distributions. We demonstrate how NIVA unifies preexisting sequence-to-sequence trajectory prediction models and emerging closed-loop simulation models trained on Next-token Prediction (NTP) from a Bayesian inference perspective. Experiments on the Waymo Open Motion Dataset demonstrate that NIVA attains competitive performance compared to the existing method while providing embellishing control over intentions and driving styles.

Multimodal large language models (MLLMs) have shown significant potential in enhancing driving scene understanding and visual question answering (VQA) through advanced logical reasoning capabilities. These tasks support driving action generation and explanation, especially in end-to-end autonomous driving applications. However, deploying these models poses a significant challenge due to their substantial parameter sizes and computational demands, which often exceed onboard computational limits. A key limitation stems from the large number of visual tokens needed to capture detailed, long-context visual information, resulting in increased latency and memory use. To address this, we propose Video Token Sparsification (VTS), a novel approach that leverages redundancy in consecutive video frames to reduce visual tokens while preserving critical information. VTS employs a lightweight CNN-based model to identify key frames and prune less informative tokens, mitigating hallucinations and boosting inference throughput without performance loss. Comprehensive experiments on the LingoQA and DRAMA benchmarks show that VTS achieves up to a 33% improvement in inference throughput and a 28% reduction in memory usage compared to baselines, maintaining comparable performance.

Understanding a driver's behavior and intentions is important for potential risk assessment and early accident prevention. Safety and driver assistance systems can be tailored to individual drivers' behavior, significantly enhancing their effectiveness. However, existing datasets are limited in describing and explaining general vehicle movements based on external visual evidence. This paper introduces a benchmark, PDB-Eval, for a detailed understanding of Personalized Driver Behavior, and aligning Large Multimodal Models (MLLMs) with driving comprehension and reasoning. Our benchmark consists of two main components, PDB-X and PDBQA. PDB-X can evaluate MLLMs' understanding of temporal driving scenes. Our dataset is designed to find valid visual evidence from the external view to explain the driver's behavior from the internal view. To align MLLMs' reasoning abilities with driving tasks, we propose PDB-QA as a visual explanation question-answering task for MLLM instruction fine-tuning. As a generic learning task for generative models like MLLMs, PDB-QA can bridge the domain gap without harming MLLMs' generalizability. Our evaluation indicates that fine-tuning MLLMs on fine-grained descriptions and explanations can effectively bridge the gap between MLLMs and the driving domain, which improves zero-shot performance on question-answering tasks by up to 73.2%. We further evaluate the MLLMs fine-tuned on PDB-X in Brain4Cars' intention prediction and AIDE's recognition tasks. We observe up to 12.5% performance improvements on the turn intention prediction task in Brain4Cars, and consistent performance improvements up to 11.0% on all tasks in AIDE.

Autonomous driving (AD) has made significant strides in recent years. However, existing frameworks struggle to interpret and execute spontaneous user instructions, such as "overtake the car ahead.” Large Language Models (LLMs) have demonstrated impressive reasoning capabilities showing potential to bridge this gap. In this paper, we present LaMPilot, a novel framework that integrates LLMs into AD systems, enabling them to follow user instructions by generating code that leverages established functional primitives. We also introduce LaMPilot-Bench, the first bench-mark dataset specifically designed to quantitatively evaluate the efficacy of language model programs in AD. Adopting the LaMPilot framework, we conduct extensive experiments to assess the performance of off-the-shelf LLMs on LaMPilot-Bench. Our results demonstrate the potential of LLMs in handling diverse driving scenarios and following user instructions in driving. To facilitate further research in this area, we release our code and data at GitHub.com/PurdueDigitalTwin/LaMPilot.

In-vehicle edge computing is a much anticipated paradigm to serve ever-increasing computation demands originated from the ego vehicle, such as passenger entertainments. In this paper, we explore the unique idea of crowdsourcing passing-by vehicles to augment computing of the ego vehicle. The challenges lie in the high dynamics of passing-by vehicles, time-correlated task computation, and the stringent requirement of computing reliability for individual user tasks. To this end, we formulate an optimization problem to minimize the end-to-end latency by optimizing the task assignment and resource allocation of user tasks. To address the complex problem, we propose a new algorithm (named CAVE) with multiple key designs. First, we reformulate the original problem into two subproblems while incorporating not only incoming but also in-progress tasks. Second, we solve the task assignment subproblem with reliability constraints by using particle swarm optimization with the adaptive barrier function. Third, we solve the resource allocation subproblem by deriving the optimal allocation with Karush–Kuhn–Tucker (KKT) condition. We build an end-to-end network and compute simulator and conduct extensive simulation to evaluate the performance of the proposed algorithm. Simulation results show that, our CAVE algorithm reduces more than 15% end-to-end latency than state-of-the-art solutions, without degrading the reliability performance.

Connected and autonomous vehicles (CAVs) rely heavily upon time-sensitive information update services to ensure the safety of people and assets, and satisfactory entertainment applications. Therefore, the freshness of information is a crucial performance metric for CAV services. However, information from roadside sensors and nearby vehicles can get delayed in transmission due to the high mobility of vehicles. Our research shows that a CAV's relative distance and speed play an essential role in determining the Age-of- Information (AoI). With an increase in AoI, incremental service aggregation issues are observed with out-of-sequence information updates, which hampers the performance of low-latency applications in CAVs. In this paper, we propose a novel AoI-based service aggregation method for CAVs, which can process the information updates according to their update cycles. First, the AoI for sensors and vehicles is modeled, and a predictive AoI system is designed. Then, to reduce the overall service aggregation time and computational load, intervals are used for periodic AoI prediction, and information sources are clustered based on the AoI value. Finally, the system aggregates services for CAV applications using the predicted AoI. We evaluate the system performance based on data sequencing success rate (DSSR), and overall system latency. Lastly, we compare the performance of our proposed system with three other state-of-the-art methods. The evaluation and comparison results show that our proposed predictive AoI-based service aggregation system maintains satisfactory latency and DSSR for CAV applications and outperforms other existing methods.

The rapid integration of augmented reality (AR) and virtual reality (VR) technologies into contemporary automotive development has led to unprecedented opportunities and challenges. This work addresses the integration of edge computing and AR/VR applications within connected autonomous vehicles, focusing on the pivotal role of object detection. The edge-assisted object detection problem is formulated as a constrained optimization problem, aiming to minimize the adverse effects on the object detection process. To solve the problem, we introduce an innovative edge-assisted algorithm, transmitting live camera frames to an edge server for detailed processing. Only essential detection data is then relayed to AR/VR devices, marking a significant advancement over existing strategies. Notable outcomes include a reduction in latency (averaging between 37.06% and 44.76%), enhanced data throughput (ranging from 27.66% to 41.18%), improved freshness loss (between 36.36% and 69.57%), and a frame loss reduction to 7.5%, surpassing baseline methods by 6.5% to 36%. These findings underscore the potential of this methodology for optimizing AR/VR applications in vehicular environments.

Concept bottleneck models have been successfully used for explainable machine learning by encoding information within the model with a set of human-defined concepts. In the context of human-assisted or autonomous driving, explainability models can help user acceptance and understanding of decisions made by the autonomous vehicle, which can be used to rationalize and explain driver or vehicle behavior. We propose a new approach using concept bottlenecks as visual features for control command predictions and explanations of user and vehicle behavior. We learn a human-understandable concept layer that we use to explain sequential driving scenes while learning vehicle control commands. This approach can then be used to determine whether a change in a preferred gap or steering commands from a human (or autonomous vehicle) is led by an external stimulus or change in preferences. We achieve competitive performance to latent visual features while gaining interpretability within our model setup.1

High-definition (HD) map is an essential building block in the autonomous driving era, which enables fine-grained environmental awareness, exact localization, and route planning. However, because HD maps include rich, multidimensional information, the volume of HD map data is enormous, making it expensive and time-consuming to transmit on vehicular networks. Therefore, in this paper, we propose a data optimization scheme for effective HD map updates in vehicular named data networking (NDN) scenarios. We formulate the HD map data optimization problem as a convex optimization problem and solve it with modified convolutional neural networks (CNNs) from YOLOX's real-time object detection system. Specifically, we modify the YOLOX object detection algorithm to detect and compress redundant pixels in local map data before transmission to the MEC server. To deploy our proposed scheme, we construct a vehicular NDN environment for data collection, processing, and transmission using the CARLA simulator and robot operating system 2 (ROS2). Extensive simulations show that our proposed scheme can significantly reduce the transmission data size and time by 48.25% - 65.78% and 46.85% - 78.84% compared with state-of-the-art HD map update techniques like RLSS, Pro-RTT, and Loss-based systems.

Artificial intelligence (AI) has enabled a new paradigm of smart applications - changing our way of living entirely. Many of these AI-enabled applications have very stringent latency requirements, especially for applications on mobile devices (e.g., smartphones, wearable devices, and vehicles). Hence, smaller and quantized deep neural network (DNN) models are developed for mobile devices, which provide faster and more energy-efficient computation for mobile AI applications. However, how AI models consume energy in a mobile device is still unexplored. Predicting the energy consumption of these models, along with their different applications, such as vision and non-vision, requires a thorough investigation of their behavior using various processing sources. In this paper, we introduce a comprehensive study of mobile AI applications considering different DNN models and processing sources, focusing on computational resource utilization, delay, and energy consumption. We measure the latency, energy consumption, and memory usage of all the models using four processing sources through extensive experiments. We explain the challenges in such investigations and how we propose to overcome them. Our study highlights important insights, such as how mobile AI behaves in different applications (vision and non-vision) using CPU, GPU, and NNAPI. Finally, we propose a novel Gaussian process regression-based general predictive energy model based on DNN structures, computation resources, and processors, which can predict the energy for each complete application cycle irrespective of device configuration and application. This study provides crucial facts and an energy prediction mechanism to the AI research community to help bring energy efficiency to mobile AI applications.

Crowdsourcing data from connected and automated vehicles (CAVs) is a cost-efficient way to achieve high-definition maps with up-to-date transient road information. Achieving the map with deterministic latency performance is, however, challenging due to the unpredictable resource competition and distributional resource demands. In this paper, we propose CoMap, a new crowdsourcing high definition (HD) map to minimize the monetary cost of network resource usage while satisfying the percentile requirement of end-to-end latency. We design a novel CROP algorithm to learn the resource demands of CAV offloading, optimize offloading decisions, and proactively allocate temporal network resources in a fully distributed manner. In particular, we create a prediction model to estimate the uncertainty of resource demands based on Bayesian neural networks and develop a utilization balancing scheme to resolve the imbalanced resource utilization in individual infrastructures. We evaluate the performance of CoMap with extensive simulations in an automotive edge computing network simulator. The results show that CoMap reduces up to 80.4% average resource usage as compared to existing solutions.

Video instance segmentation has emerged as a critical component in enabling connected vehicles to comprehend complex driving scenes, thereby facilitating navigation under various driving conditions. Recent advances focus on video-based solutions, which leverage temporal and spatial information to achieve superior performance compared to the traditional image-based approaches. However, these video-based solutions present challenges for efficient deployment at the edge due to their high computational and memory demands, making them inefficient for deployment on edge devices, such as intelligent vehicles. Furthermore, the large size of video data makes it impractical to upload to cloud servers. To address the latency challenge during on-device inference, we propose to incorporate early exits into the model. While the early exit strategy has been successful in image classification and natural language processing tasks, our study is the first to explore its application in video instance segmentation. Specifically, we incorporate early exits into the transformer-based video instance segmentation model, VisTR. Our experimental results on the YouTube-VIS dataset demonstrate that early exit can significantly speed up the inference by up to 4.83× with a minimal trade-off of only 3% in the averaged precision scores. Furthermore, our qualitative analysis confirms the satisfactory quality of the generated segmentation masks.

Cooperative perception is the key approach to augment the perception of connected and automated vehicles (CAVs) toward safe autonomous driving. However, it is challenging to achieve real-time perception sharing for hundreds of CAVs in large-scale deployment scenarios. In this paper, we propose AdaMap, a new high-scalable real-time cooperative perception system, which achieves assured percentile end-to-end latency under time-varying network dynamics. To achieve AdaMap, we design a tightly coupled data plane and control plane. In the data plane, we design a new hybrid localization module to dynamically switch between object detection and tracking, and a novel point cloud representation module to adaptively compress and reconstruct the point cloud of detected objects. In the control plane, we design a new graph-based object selection method to un-select excessive multi-viewed point clouds of objects, and a novel approximated gradient descent algorithm to optimize the representation of point clouds. We implement AdaMap on an emulation platform, including realistic vehicle and server computation and a simulated 5G network, under a 150-CAV trace collected from the CARLA simulator. The evaluation results show that, AdaMap reduces up to 49x average transmission data size at the cost of 0.37 reconstruction loss, as compared to state-of-the-art solutions, which verifies its high scalability, adaptability, and computation efficiency.

Today, deep learning optimization is primarily driven by research focused on achieving high inference accuracy and reducing latency. However, the energy efficiency aspect is often overlooked, possibly due to a lack of sustainability mindset in the field and the absence of a holistic energy dataset. In this paper, we conduct a threefold study, including energy measurement, prediction, and efficiency scoring, with an objective to foster transparency in power and energy consumption within deep learning across various edge devices. Firstly, we present a detailed, first-of-its-kind measurement study that uncovers the energy consumption characteristics of on-device deep learning. This study results in the creation of three extensive energy datasets for edge devices, covering a wide range of kernels, state-of-the-art DNN models, and popular AI applications. Secondly, we design and implement the first kernel-level energy predictors for edge devices based on our kernel-level energy dataset. Evaluation results demonstrate the ability of our predictors to provide consistent and accurate energy estimations on unseen DNN models. Lastly, we introduce two scoring metrics, PCS and IECS, developed to convert complex power and energy consumption data of an edge device into an easily understandable manner for edge device end-users. We hope our work can help shift the mindset of both end-users and the research community towards sustainability in edge computing, a principle that drives our research. Find data, code, and more up-to-date information at https://amai-gsu.github.io/DeepEn2023.

In order to enhance driving safety and identify potential hazards, next-generation intelligent vehicles will need to understand human drivers’ intentions and predict their potential maneuvers correctly. In a lane-change scenario, a driver’s head rotation measured by the in-cabin driver monitoring camera can serve as a reliable indicator to predict his/her intention. However, using a general model to predict each driver’s maneuver is not accurate, while directly sharing the personalized monitoring data to other intelligent vehicles raises the privacy concern. In this paper, we propose a clustering-based personalized federated learning framework (CPFL) to predict lane-change maneuver based on driver monitoring data. Personalization is added on top of the traditional federated learning (FL) through clustering, which separates and groups similar driving behaviors based on clustering parameters: head position threshold and average pre-lane-change preparation time. Long-Short Term Memory (LSTM) networks with different sequence lengths are deployed to predict lane changes in different clusters based on the lane-change preparation time. CPFL framework is trained and tested using the data collected from several human drivers under different driving scenarios through the Unity simulation platform. According to the results, CPFL’s average training efficiency is 7.6 times higher than the classic FedAvg approach, and CPFL also offers better adaptability to different driving behaviors than FedAvg with 4% higher accuracy, 0.2% fewer false positives, and 27.8% fewer false negatives.

Understanding the behavior of human drivers and how they interact with other drivers is crucial to develop and improve the decision-making capabilities of connected and automated vehicles (CAVs). This allows CAVs to anticipate and proactively respond to the actions of other road users in a safe and efficient manner, especially in mixed-traffic environments. Most existing studies rely on neural networks to model such interaction implicitly, and very few studies attempt to interpret the interaction. Considering its interpretability and flexibility, the Granger causality (GC) framework is widely used to understand the relationships between different agents, as well as how these relationships change over time. In this paper, we integrate the knowledge of traffic and vehicle dynamics into the neural network to learn the Granger causality and explore vehicular interaction from multi-vehicle trajectories. The proposed algorithm has been validated using both the INTERACTION dataset and field data collected in Riverside, California. The results show that our algorithm is able to address three key questions regarding vehicular interaction: 1) whether the interactions exist between/among the vehicles; 2) when the interactions occur and terminate; and 3) how strong the interactions between/among vehicles are.

Advanced Driver Assistance Systems (ADAS) are increasingly important in improving driving safety and comfort, with Adaptive Cruise Control (ACC) being one of the most widely used. However, pre-defined ACC settings may not always align with driver's preferences and habits, leading to discomfort and potential safety issues. Personalized ACC (P-ACC) has been proposed to address this problem, but most existing research uses historical driving data to imitate behaviors that conform to driver preferences, neglecting real-time driver feedback. To bridge this gap, we propose a cloud-vehicle collaborative P-ACC framework that incorporates driver feedback adaptation in real time. The framework is divided into offline and online parts. The offline component records the driver's naturalistic car-following trajectory and uses inverse reinforcement learning (IRL) to train the model on the cloud. In the online component, driver feedback is used to update the driving gap preference in real time. The model is then retrained on the cloud with driver's takeover trajectories, achieving incremental learning to better match driver's preference. Human-in-the-loop (HuiL) simulation experiments demonstrate that our proposed method significantly reduces driver intervention in automatic control systems by up to 62.8%. By incorporating real-time driver feedback, our approach enhances the comfort and safety of P-ACC, providing a personalized and adaptable driving experience.

Connected and automated vehicles (CAVs) are supposed to share the road with human-driven vehicles (HDVs) in a foreseeable future. Therefore, considering the mixed traffic environment is more pragmatic, as the well-planned operation of CAVs may be interrupted by HDVs. In the circumstance that human behaviors have significant impacts, CAVs need to understand HDV behaviors to make safe actions. In this study, we develop a driver digital twin (DDT) for the online prediction of personalized lane-change behavior, allowing CAVs to predict surrounding vehicles’ behaviors with the help of the digital twin technology. DDT is deployed on a vehicle-edge–cloud architecture, where the cloud server models the driver behavior for each HDV based on the historical naturalistic driving data, while the edge server processes the real-time data from each driver with his/her digital twin on the cloud to predict the personalized lane-change maneuver. The proposed system is first evaluated on a human-in-the-loop co-simulation platform, and then in a field implementation with three passenger vehicles driving along an on/off ramp segment connecting to the edge server and cloud through the 4G/LTE cellular network. The lane-change intention can be recognized in 6 s on average before the vehicle crosses the lane separation line, and the Mean Euclidean Distance between the predicted trajectory and GPS ground truth is 1.03 m within a 4-s prediction window. Compared to the general model, using a personalized model can improve prediction accuracy by 27.8%. The demonstration video of the proposed system can be watched athttps://youtu.be/5cbsabgIOdM.

Highly-dynamic (HD) map is an indispensable building block in the future of autonomous driving, allowing for fine-grained environmental awareness, precise localization, and route planning. However, since HD maps include rich, multidimensional information, the volume of HD map data is substantial and cannot be transmitted frequently by several vehicles over vehicular networks in real-time. Therefore, in this paper, we propose a data source selection scheme for effective HD map transmissions in vehicular named data networking (NDN) scenarios. To achieve our goal, we created a vehicular NDN environment for data collection, processing, and transmission using the CARLA simulator and robot operating system 2 (ROS2). Next, due to our vehicular NDN’s dynamic and complex nature, we formulate the data source selection problem as a Markov decision process (MDP) and solve it using a reinforcement learning approach. For simplicity, we termed our proposed scheme data source optimization with reinforcement learning (DSORL), which selects suitable vehicles for HD map data transmission to MEC servers. The experiment results indicate that our suggested method outperformed existing baseline schemes, such as RLSS, Pro-RTT, and HDM-RTT, across all performance criteria in the evaluation. For instance, the system throughput increases by$65\%-72.68\%$compared to other baseline systems. Similarly, the proposed approach can minimize packet loss rate, data size, and transmission time by up to 60.6%, 77.5%, and 54.1%, respectively.

In the foreseeable future, connected and auto-mated vehicles (CAVs) and human-driven vehicles will share the road networks together. In such a mixed traffic environment, CAVs need to understand and predict maneuvers of surrounding vehicles for safer and more efficient interactions, especially when human drivers bring in a wide range of uncertainties. In this paper, we propose a learning-based lane-change prediction algorithm that considers the driving behaviors of the target human driver. To provide accurate maneuver prediction, we adopt a hierarchical structure that seamlessly seals both the lane-change decision prediction and the vehicle trajectory pre-diction together. Specifically, we propose a lane-change decision prediction method based on a Long-Short Term Memory (LSTM) network, and a trajectories prediction considering driver preference and vehicular interactions based on Inverse Reinforcement Learning (IRL). To validate the performance of the proposed methodology, a case study of an on-ramp merging scenario is conducted on a uniquely built human-in-the-loop simulation platform that can provide an immersive driving environment, collect data of lane-change behaviors, and test drivers' reactions to the prediction results in real time. It is shown in the simulation results that we can predict the lane-change decision 3 seconds before the vehicle crosses the line to another lane, and the Mean Euclidean Distance between the predicted trajectory and ground truth is 0.39 meters within a 4-second prediction window.

Driving automation is gradually replacing human driving maneuvers in different applications such as adaptive cruise control and lane keeping. However, contemporary driving automation applications based on expert systems or prede-fined control strategies are not in line with individual human driver's preference. To overcome this problem, we propose a Personalized Adaptive Cruise Control (P-ACC) system that can learn the driver's car-following preferences from historical data using model-based maximum entropy Inverse Reinforcement Learning (IRL). Once activated in real-time, the P-ACC system first classifies the driver type and the weather type (at that moment). The vehicle is then controlled using the pre-trained IRL model on the cloud of the associated class. The personalized IRL model on the cloud will be updated as more human driving data is collected from various scenarios. Numerical simulation with real-world naturalistic driving data shows that, the accuracy of reproducing the real-world driving profile improves up to 30.1% in terms of speed and 36.5% in terms of distance gap, when P-ACC is compared with the Intelligent Driver Model (IDM). Game engine-based human-in-the-loop simulation demonstrates that, the takeover frequency of the driver during the usage of P-ACC decreases up to 93.4%, compared with that during the usage of IDM-based ACC.

A Mobility Digital Twin is an emerging implementation of Digital Twin in the transportation domain, and has been attracting extensive attention from both industry and academia. Although a few research have been conducted on the mobility digital twin, there is no systematic work with an end-to-end digital twin model construction framework. In this paper, we propose an end-to-end system framework, including sensory data collection, offloading, and processing, that aims to facilitate a high-fidelity and real-time digital twin model construction for connected and automated vehicles. Additionally, preliminary experiments are conducted to demonstrate our research motivation and to guide the future system framework design.

A Digital Twin is a digital replica of a living or nonliving physical entity, and this emerging technology attracted extensive attention from different industries during the past decade. Although a few Digital Twin studies have been conducted in the transportation domain very recently, there is no systematic research with a holistic framework connecting various mobility entities together. In this study, a mobility digital twin (MDT) framework is developed, which is defined as an artificial intelligence (AI)-based data-driven cloud–edge–device framework for mobility services. This MDT consists of three building blocks in the physical space (namely,Human,Vehicle, andTraffic), and their associated Digital Twins in the digital space. An example cloud–edge architecture is built with Amazon Web Services (AWS) to accommodate the proposed MDT framework and to fulfill its digital functionalities of storage, modeling, learning, simulation, and prediction. A case study of the personalized adaptive cruise control (P-ACC) system is conducted, which integrates the key microservices of all three digital building blocks of the MDT framework: 1) theHuman Digital Twinwith user management and driver type classification; 2) theVehicle Digital Twinwith cloud-based advanced driver-assistance systems (ADAS); and 3) theTraffic Digital Twinwith traffic flow monitoring and variable speed limit. Future challenges of the proposed MDT framework are discussed toward the end of the article, including standardization, AI for computing, public or private cloud service, and network heterogeneity.

Recent work has considered personalized route planning based on user profiles, but none of it accounts for human trust. We argue that human trust is an important factor to consider when planning routes for automated vehicles. This article presents a trust-based route-planning approach for automated vehicles. We formalize the human-vehicle interaction as a partially observable Markov decision process (POMDP) and model trust as a partially observable state variable of the POMDP, representing the human’s hidden mental state. We build data-driven models of human trust dynamics and takeover decisions, which are incorporated in the POMDP framework, using data collected from an online user study with 100 participants on the Amazon Mechanical Turk platform. We compute optimal routes for automated vehicles by solving optimal policies in the POMDP planning and evaluate the resulting routes via human subject experiments with 22 participants on a driving simulator. The experimental results show that participants taking the trust-based route generally reported more positive responses in the after-driving survey than those taking the baseline (trust-free) route. In addition, we analyze the trade-offs between multiple planning objectives (e.g., trust, distance, energy consumption) via multi-objective optimization of the POMDP. We also identify a set of open issues and implications for real-world deployment of the proposed approach in automated vehicles.

Ramp merging is considered as one of the most difficult driving scenarios due to the chaotic nature in both longitudinal and lateral driver behaviors (namely lack of effective coordination) in the merging area. In this study, we have designed a cooperative ramp merging system for connected vehicles, allowing merging vehicles to cooperate with others prior to arriving at the merging zone. Different from most of the existing studies that utilize dedicated short-range communication, we adopt a Digital Twin approach based on vehicle-to-cloud communication. On-board devices upload the data to the cloud server through the 4G/LTE cellular network. The server creates Digital Twins of vehicles and drivers whose parameters are synchronized in real time with their counterparts in the physical world, processes the data with the proposed models in the digital world, and sends advisory information back to the vehicles and drivers in the physical world. A real-world field implementation has been conducted in Riverside, California, with three passenger vehicles. The results show the proposed system addresses the issues of safety and environmental sustainability with an acceptable communication delay, compared to the baseline scenario where no advisory information is provided during the merging process.

Advanced driver-assistance systems (ADAS) have matured over the past few decades with the dedication to enhance user experience and gain a wider market penetration. However, personalization features, as an approach to make the current technologies more acceptable and trustworthy for users, have been gaining momentum only very recently. In this work, we aim to learn personalized longitudinal driving behaviors via a Gaussian Process (GP) model. The proposed method learns from individual driver’s naturalistic car-following behavior, and outputs a desired acceleration profile that suits the driver’s preference. The learned model, together with a predictive safety filter that prevents rear-end collision, is used as a personalized adaptive cruise control (PACC) system. Numerical experiments show that GP-based PACC (GP-PACC) can almost exactly reproduce the driving styles of an intelligent driver model. Additionally, GP-PACC is further validated by human-in-the-loop experiments on the Unity game engine-based driving simulator. Trips driven by GP-PACC and two other baseline ACC algorithms with driver override rates are recorded and compared. Results show that on average, GP-PACC reduces the human override duration by 60% and 85% as compared to two widely-used ACC models, respectively, which shows the great potential of GP-PACC in improving driving comfort and overall user experience.

Ramp merging is considered to be one of the major causes of traffic accidents and congestion due to its inherent chaotic nature. With the development of the connected and automated vehicle (CAV) technology, CAVs can conduct cooperative merging using communication, and can also handle complicated situations even with legacy vehicles. In this article, a game theory-based ramp merging strategy has been developed for the optimal merging coordination of CAVs in mixed traffic, which can determine the dynamic merging sequence and corresponding longitudinal/lateral control. This strategy improves the safety and efficiency of the merging process by ensuring a safe intervehicle distance and harmonizing the speeds of CAVs in the traffic stream. To verify the proposed strategy, mixed traffic simulation runs under different penetration rates and different congestion levels have been carried out on an innovative Unity-SUMO integrated platform, which connects a game engine-based driving simulator with a state-of-the-art microscopic traffic simulator. The results show that the average speed of traffic flow can be increased up to 210%, while the fuel consumption can be reduced up to 53.9%. In addition, the driving volatility can be stabilized to a level with 0% extreme values.

With the rapid development of intelligent vehicles and Advanced Driving Assistance Systems (ADAS), a mixed level of human driver engagements is involved in the transportation system. Visual guidance for drivers is essential under this situation to prevent potential risks. To advance the development of visual guidance systems, we introduce a novel sensor fusion methodology, integrating camera image and Digital Twin knowledge from the cloud. Target vehicle bounding box is drawn and matched by combining results of object detector running on ego vehicle and position information from the cloud. The best matching result, with a 79.2% accuracy under 0.7 Intersection over Union (IoU) threshold, is obtained with depth image served as an additional feature source. Game engine-based simulation results also reveal that the visual guidance system could improve driving safety significantly cooperate with the cloud Digital Twin system.

Behavior prediction plays an essential role in both autonomous driving systems and Advanced Driver Assistance Systems (ADAS), since it enhances vehicle's awareness of the imminent hazards in the surrounding environment. Many existing lane change prediction models take as input lateral or angle information and make short-term (<; 5 seconds) maneuver predictions. In this study, we propose a longer-term (5~10 seconds) prediction model without any lateral or angle information. Three prediction models are introduced, including a logistic regression model, a multilayer perceptron (MLP) model, and a recurrent neural network (RNN) model, and their performances are compared by using the real-world NGSIM dataset. To properly label the trajectory data, this study proposes a new time-window labeling scheme by adding a time gap between positive and negative samples. Two approaches are also proposed to address the unstable prediction issue, where the aggressive approach propagates each positive prediction for certain seconds, while the conservative approach adopts a roll-window average to smooth the prediction. Evaluation results show that the developed prediction model is able to capture 75% of real lane change maneuvers with an average advanced prediction time of 8.05 seconds.

With the development of advanced communication technology, connected vehicles become increasingly popular in our transportation systems, which can conduct cooperative maneuvers with each other as well as road entities through vehicle-to-everything communication. A lot of research interests have been drawn to other building blocks of a connected vehicle system, such as communication, planning, and control. However, less research studies were focused on the human-machine cooperation and interface, namely how to visualize the guidance information to the driver as an advanced driver-assistance system (ADAS). In this study, we propose an augmented reality (AR)-based ADAS, which visualizes the guidance information calculated cooperatively by multiple connected vehicles. An unsignalized intersection scenario is adopted as the use case of this system, where the driver can drive the connected vehicle crossing the intersection under the AR guidance, without any full stop at the intersection. A simulation environment is built in Unity game engine based on the road network of San Francisco, and human-in-the-loop (HITL) simulation is conducted to validate the effectiveness of our proposed system regarding travel time and energy consumption.

Digital twin, an emerging representation of cyberphysical systems, has attracted increasing attentions very recently. It opens the way to real-time monitoring and synchronization of real-world activities with the virtual counterparts. In this study, we develop a digital twin paradigm using an advanced driver assistance system (ADAS) for connected vehicles. By leveraging vehicle-to-cloud (V2C) communication, on-board devices can upload the data to the server through cellular network. The server creates a virtual world based on the received data, processes them with the proposed models, and sends them back to the connected vehicles. Drivers can benefit from this V2C based ADAS, even if all computations are conducted on the cloud. The cooperative ramp merging case study is conducted, and the field implementation results show the proposed digital twin framework can benefit the transportation systems regarding mobility and environmental sustainability with acceptable communication delays and packet losses.

The success of Intelligent Driver Assistance (IDA) depends on the system's ability to accurately model the state of traffic surrounding the ego vehicle and predict driving behavior of the surrounding vehicles in order to help the ego driver make the best informed decisions in real-time. The ability to predict acceleration behavior is crucial as a first step towards modeling traffic patterns. In this paper, we show that Long Short-Term Memory (LSTM) neural networks are capable of producing acceleration distributions from which accurate future acceleration values can be sampled. Furthermore, state values calculated from these acceleration predictions are used as input for future predictions, showing that these networks are capable of generating realistic simulated vehicle trajectories over short prediction horizons.

A growing number of vehicles are equipped with Vehicle-to-vehicle (V2V) communication modules (e.g., Dedicated Short Range Communications) that allow them to exchange messages over the network. The V2V communication is expected to improve the road safety by overcoming limitation of conventional Advanced Driver Assistance Systems (ADASs). For a safe feature using V2V communication-based applications, it is essential to identify sender vehicles since V2V communication is typically implemented using a broadcast mechanism. Especially here, our focus is to correctly determine whether the preceding vehicle is the sender of the received message or not in order to realize cooperative driving such as platooning. Vehicle location information obtained by an onboard GPS module is typically used for the identification. However, the GPS module often provides wrong location information due to the limited accuracy in a certain environment such as an urban road surrounded by tall buildings. To prevent this GPS error from causing misidentifications, we propose a novel method which additionally uses shared ranging sensor data and behavioral control of the ego vehicle. Simulation result shows that our proposed method successfully reduces the number of misidentifications by 64 % compared with a method which fully depends on GPS information.

Virtual platform prototypes are widely utilized to enable early system-level design space exploration. Accurate power models for hardware components at high levels of abstraction are needed to enable system-level power analysis and optimization. However, the limited observability of third party IPs renders traditional power modeling methods challenging and inaccurate. In this paper, we present a novel approach for extending behavioral models of black-box hardware IPs with an accurate power estimate. We leverage state-of-the-art-machine learning techniques to synthesize an abstract power model. Our model uses input and output history to track data-dependent pipeline behavior. Furthermore, we introduce a specialized ensemble learning that is composed out of individually selected cycle-by-cycle models to reduce overall complexity and further increase estimation accuracy. Results of applying our approach to various industrial-strength design examples shows that our models predict average power consumption to within 3% of a commercial gate-level power estimation tool, all while running several orders of magnitude faster.

With the prevalence of System-on-Chips there is a growing need for automation and acceleration of the design process. A classical approach is to take a C/C++ specification of the application, convert it to a SystemC (or equivalent) description of hardware implementing this application, and perform successive refinement of the description to improve various design metrics. In this work, we present an automated SystemC generation and design space exploration flow alleviating several productivity and design time issues encountered in the current design process. We first automatically convert a subset of C/C++, namely affine program regions, into a full SystemC description through polyhedral model-based techniques while performing powerful data locality and parallelism transformations. We then leverage key properties of affine computations to design a fast and accurate latency and power characterization flow. Using this flow, we build analytical models of power and performance that can effectively prune away a large amount of inferior design points very fast and generate Pareto-optimal solution points. Experimental results show that (1) our SystemC models can evaluate system performance and power that is only 0.57% and 5.04% away from gate-level evaluation results, respectively; (2) our latency and power analytical models are 3.24% and 5.31% away from the actual Pareto points generated by SystemC simulation, with 2091x faster design-space exploration time on average. The generated Pareto-optimal points provide effective low-power design solutions given different latency constraints.

Our principal motivation is to reduce the energy consumption of display subsystems in mobile devices by introducing a hybrid frame buffer architecture into the platform. We observed that display contents on a screen are quite static for certain mobile workloads, such as web browsing. As a result, data reading from the display frame is much more frequent than the writing of new data onto the frame buffer, a state we refer to as read dominance. Based on this observation, we propose a hybrid frame buffer architecture that exploits the display contents' read-dominant property to improve the energy efficiency of display subsystems. Specifically, we employ two memory types: DRAM and Phase-Change Memory (PCM), in the display frame buffer to exploit their different read/write energy characteristics. We also present an analysis of the energy efficiency of the hybrid frame buffer based on our display content and energy consumption models. Our evaluation results show that the proposed hybrid frame buffer reduces frame buffer energy consumption by up to 43%, compared to the conventional DRAM-only frame buffer.

Power consumption of the display subsytem has been a relatively less explored area compared to other components of a mobile device including computing, storage, and networking units, although the former often constitutes one of the most power-hungry portions of the system. Typical applications on a mobile device such as Web browsing and text editing tend to have rather static image content; each frame hardly changes from the previous one. Efficiently detecting and handling no-motion scenarios is thus critical to extend the battery life. This paper focuses on image change detection. We propose to use checksum to detect image changes. Specifically, CRC hardware is used to optimize the power consumption of (1) refresh of a local display and (2) data compression for wireless remote display. Compared with a traditional, pixel-by-pixel comparison approach, using checksum for image change detection is not only fast, but also reduces accesses to the frame buffer, resulting in significant power savings. We have built a FPGA prototype to verify that CRC can capture image changes well enough to ensure a ¿visually lossless¿ quality.

Many digital signal processing algorithms are first developed in floating point and later mapped into fixed point for digital hardware implementation. During this mapping, wordlengths are searched to minimize total hardware cost and maximize system performance. Complexity and distortion measures have been separately researched for optimum wordlength selection. This paper proposes a complexity-and-distortion measure (CDM) method that combines these two measures. The CDM method trades off these two measures using a weighting factor. The proposed method is applied to wordlength design of a fixed broadband wireless demodulator. For this case study, the proposed method finds the optimal solution in one-third the time that exhaustive search takes. The contributions of this paper are: (1) a generalization of search methods based on complexity or distortion measures; (2) a framework of automatic wordlength optimization; and (3) a wireless demodulator case study.