We introduce ET-Former, a novel end-to-end algorithm for semantic scene completion using a single monocular camera. Our approach generates a semantic occupancy map from single RGB observation while simultaneously providing uncertainty estimates for semantic predictions. By designing a triplane-based deformable attention mechanism, our approach improves geometric understanding of the scene than other SOTA approaches and reduces noise in semantic predictions. Additionally, through the use of a Conditional Variational AutoEncoder (CVAE), we estimate the uncertainties of these predictions. The generated semantic and uncertainty maps will help formulate navigation strategies that facilitate safe and permissible decision making in the future. Evaluated on the Semantic-KITTI dataset, ET-Former achieves the highest Intersection over Union (IoU) and mean IoU (mIoU) scores while maintaining the lowest GPU memory usage, surpassing state-of-the-art (SOTA) methods. It improves the SOTA scores of IoU from 44.71 to 51.49 and mIoU from 15.04 to 16.30 on SeamnticKITTI test, with a notably low training memory consumption of 10.9 GB, achieving at least a 25% reduction compared to previous methods. Project page: https://github.com/amazon-science/ET-Former.

We present VLPG-Nav, a visual language navigation method for guiding robots to specified objects within household scenes. Unlike existing methods primarily focused on navigating the robot toward objects, our approach considers the additional challenge of centering the object within the robot’s camera view. Our method builds a visual language pose graph (VLPG) that functions as a spatial map of VL embeddings. Given an open-vocabulary object query, we plan a viewpoint for object navigation using the VLPG. Despite navigating to the viewpoint, real-world challenges such as object occlusion, displacement, and the robot’s localization errors can prevent visibility. We build an object localization probability map that leverages the robot’s current observations and prior VLPG. When the object is not visible, the probability map is updated, and an alternate viewpoint is computed. In addition, we propose an object-centering formulation that locally adjusts the robot’s pose to center the object in the camera view. We evaluate the effectiveness of our approach through simulations and real-world experiments, evaluating its ability to successfully view and center the object within the camera’s field of view. VLPG-Nav demonstrates improved performance in locating the object, navigating around occlusions, and centering the object within the robot’s camera view, outperforming selected baselines in the evaluation settings.

One key challenge for multi-task Reinforcement learning (RL) in practice is the absence of task specifications. Robust RL has been applied to deal with task ambiguity but may result in over-conservative policies. To balance the worst-case (robustness) and average performance, we propose Group Distributionally Robust Markov Decision Process (GDR-MDP), a flexible hierarchical MDP formulation that encodes task groups via a latent mixture model. GDR-MDP identifies the optimal policy that maximizes the expected return under the worst-possible qualified belief over task groups within an ambiguity set. We rigorously show that GDR-MDP’s hierarchical structure improves distributional robustness by adding regularization to the worst possible outcomes. We then develop deep RL algorithms for GDR-MDP for both value-based and policy-based RL methods. Extensive experiments on Box2D control tasks, MuJoCo benchmarks, and Google football platforms show that our algorithms outperform classic robust training algorithms across diverse environments in terms of robustness under belief uncertainties. Demos are available on our project page (https://sites.google.com/view/gdr-rl/home).

Active perception describes a broad class of techniques that couple planning and perception systems to move the robot in a way to give the robot more information about the environment. In most robotic systems, perception is typically independent of motion planning. For example, traditional object detection is passive: it operates only on the images it receives. However, we have a chance to improve the results if we allow planning to consume detection signals and move the robot to collect views that maximize the quality of the results. In this paper, we use reinforcement learning (RL) methods to control the robot in order to obtain images that maximize the detection quality. Specifically, we propose using a Decision Transformer with online fine-tuning, which first optimizes the policy with a pre-collected expert dataset and then improves the learned policy by exploring better solutions in the environment. We evaluate the performance of proposed method on an interactive dataset collected from an indoor scenario simulator. Experimental results demonstrate that our method outperforms all baselines, including expert policy and pure offline RL methods. We also provide exhaustive analyses of the reward distribution and observation space.

Object perception plays a fundamental role in Cooperative Driving Automation (CDA) which is regarded as a revolutionary promoter for next-generation transportation systems. However, the vehicle-based perception may suffer from the limited sensing range and occlusion as well as low penetration rates in connectivity. In this paper, we propose Cyber Mobility Mirror (CMM), a next-generation real-world object perception system for 3D object detection, tracking, localization, and reconstruction, to explore the potential of roadside sensors for enabling CDA in the real world. The CMM system consists of six main components: i) the data pre-processor to retrieve and preprocess the raw data; ii) the roadside 3D object detector to generate 3D detection results; iii) the multi-object tracker to identify detected objects; iv) the global locator to generate geo-localization information; v) the mobile-edge-cloud-based communicator to transmit perception information to equipped vehicles, and vi) the onboard advisor to reconstruct and display the real-time traffic conditions. An automatic perception evaluation approach is proposed to support the assessment of data-driven models without human-labeling requirements and a CMM field-operational system is deployed at a real-world intersection to assess the performance of the CMM. Results from field tests demonstrate that our CMM prototype system can achieve 96.99% precision and 83.62% recall for detection and 73.55% ID-recall for tracking. High-fidelity real-time traffic conditions (at the object level) can be geo-localized with a root-mean-square error (RMSE) of$0.69m$and$0.33m$for lateral and longitudinal direction, respectively, and displayed on the GUI of the equipped vehicle with a frequency of$3-4 Hz$.

Evaluating rare but high-stakes events is one of the main challenges in obtaining reliable reinforcement learning policies, especially in large or infinite state/action spaces where limited scalability dictates a prohibitively large number of testing iterations. On the other hand, a biased or inaccurate policy evaluation in a safety-critical system could potentially cause unexpected catastrophic failures during deployment. This paper proposes the Accelerated Policy Evaluation (APE) method, which simultaneously uncovers rare events and estimates the rare event probability in Markov decision processes. The APE method treats the environment nature as an adversarial agent and learns towards, through adaptive importance sampling, the zero-variance sampling distribution for the policy evaluation. Moreover, APE is scalable to large discrete or continuous spaces by incorporating function approximators. We investigate the convergence property of APE in the tabular setting. Our empirical studies show that APE can estimate the rare event probability with a smaller bias while only using orders of magnitude fewer samples than baselines in multi-agent and single-agent environments.

Object detection and tracking play a fundamental role in enabling Cooperative Driving Automation (CDA), which is regarded as the revolutionary solution to addressing safety, mobility, and sustainability issues of contemporary transportation systems. Although current computer vision technologies can provide satisfactory object detection results in occlusion-free scenarios, the perception performance of onboard sensors is inevitably limited by the range and occlusion. Owing to the flexible location and pose for sensor installation, infrastructure-based detection, and tracking systems can enhance the perception capability of connected vehicles; as such, they have quickly become a popular research topic. In this survey paper, we review the research progress for infrastructure-based object detection and tracking systems. Architectures of roadside perception systems based on different types of sensors are reviewed to show a high-level description of the workflows for infrastructure-based perception systems. Roadside sensors and different perception methodologies are reviewed and analyzed with detailed literature to provide a low-level explanation for specific methods followed by Datasets and Simulators to draw an overall landscape of infrastructure-based object detection and tracking methods. We highlight current opportunities, open problems, and anticipated future trends.

Multi-agent trajectory prediction is a challenging task because of the uncertainty of agents’ behaviors, interactions between agents, complex road geometry in urban environments, and imperfect/noisy agent histories. Although accurate prediction results are critical for safe and reliable intelligent driving applications (e.g., decision making, motion planning), some other applications may prefer light-weight and computation-efficient trajectory prediction models to handle dynamically changed environments. In this work, we propose a multi-agent, multi-modal Graph Attention Isomorphism Network (GAIN) based trajectory prediction framework to effectively understand and aggregate long-term interactions across agents. We also take the model complexity and computation efficiency into consideration. Experiments on both pedestrian and vehicle datasets demonstrated the effectiveness of our proposed method.

Environment perception including detection, classification, tracking, and motion prediction are key enablers for automated driving systems and intelligent transportation applications. Fueled by the advances in sensing technologies and machine learning techniques, LiDAR-based sensing systems have become a promising solution. The current challenges of this solution are how to effectively combine different perception tasks into a single backbone and how to efficiently learn the spatiotemporal features directly from point cloud sequences. In this research, we propose a novel spatiotemporal attention network based on a transformer self-attention mechanism for joint semantic segmentation and motion prediction within a point cloud at the voxel level. The network is trained to simultaneously outputs the voxel level class and predicted motion by learning directly from a sequence of point cloud datasets. The proposed backbone includes both a temporal attention module (TAM) and a spatial attention module (SAM) to learn and extract the complex spatiotemporal features. This approach has been evaluated with the nuScenes dataset, and promising performance has been achieved.

Detailed road features like lane markers and stop bars are crucial for many recent Intelligent Transportation System (ITS) applications, especially for advanced driving assistant systems or autonomous vehicles. In this paper, a data-driven method is proposed to identify intersection areas and map stop bar positions without prior knowledge of road information. The proposed method includes 1) a novel and efficient approach to identify intersections by analyzing the entropy of vehicles’ moving directions; and 2) a statistical model for estimating the number, coordinates, and directions of stop bars by evaluating the upstream vehicles’ stopping locations. By applying the method to real-world vehicle positioning data collected at Ann Arbor, its applicability and robustness to handle data at an urban regional scale (a 1.2 km by 2 km rectangular area) are proven. The accuracy of intersection identification is 95.7% for trajectory covered regions. For stop bar positioning, the mean and standard deviation of the errors are 0.27 m and 0.32 m respectively, which satisfy most of the mobility and eco-driving connected and automated vehicle applications such as eco-approach and departure at signalized intersections.

Using connected vehicle technology, a number of eco-approach and departure (EAD) strategies have been designed to guide vehicles through signalized intersections in an eco-friendly way. Most of the existing EAD applications have been developed and tested in traffic-free scenarios or in a fully connected environment, where the presence and behavior of all surrounding vehicles are detectable. In this paper, we describe a prediction-based EAD strategy that can be applied toward more realistic scenarios, where the surrounding vehicles can be either a connected or non-connected. Unlike highway scenarios, predicting speed trajectories along signalized corridors is much more challenging due to disturbances from signals, traffic queues, and pedestrians. Based on vehicle activity data available via inter-vehicle communication or onboard sensing (e.g., by radar), we evaluate three state-of-the-art nonlinear regression models to perform short-term speed forecasting of the preceding vehicle. It turns out radial basis function neural network outperformed both Gaussian process and multi-layer perceptron network in terms of prediction accuracy and computational efficiency. Using signal phase and timing information and the predicted state of the preceding vehicle, our prediction-based EAD algorithm achieved better fuel economy and emissions reduction in urban traffic and queues at intersections. Results from the numerical simulation using the next generation simulation data set show that the proposed prediction-based EAD system achieve 4.0% energy savings and 4.0% - 41.7% pollutant emission reduction compared with a conventional car following strategy. Prediction-based EAD saves 1.9% energy and reduces criteria pollutant emissions by 1.9% - 33.4% compared with an existing EAD algorithm without prediction in urban traffic.

In recent years, eco-friendly driving or ecodriving technologies are being developed to assist human drivers to achieve maximum fuel/energy efficiency in different driving conditions. Enhanced by V2X wireless communications, connected ecodriving is expected to be very promising in reducing transportation-related fossil fuel consumption as well as pollutant emissions. Besides, the deployment of electric vehicles (EVs) also has great potential in reducing greenhouse gas emissions due to the use of batteries as the sole energy source. Although recent research shows that significant energy savings can be achieved with the aid of ecodriving systems in real-world driving, there have been very few research efforts that consider human driver error, especially for electric vehicle (EV) driving. In this paper, a connected cooperative ecodriving system for energy-efficient driving that considers human driver error is designed and evaluated with an EV energy consumption model. Real-world driving data were collected and used to evaluate system performance in terms of energy consumption. The simulation and numerical analysis shows that an average of 12% energy savings can be achieved by the proposed system that considers human driver error comparing with the conventional ecodriving system without considering driver error.

To mitigate air pollution problems and reduce greenhouse gas emissions (GHG), plug-in hybrid electric vehicles (PHEV) have been developed to achieve higher fuel efficiency. The Energy Management System (EMS) is a very important component of a PHEV in achieving better fuel economy and it is a very active research area. So far, most of the existing EMS strategies just simple follow predefined rules that are not adaptive to changing driving conditions; other strategies as starting to incorporate accurate prediction of future traffic conditions. In this study, a deep reinforcement learning based PHEV energy management system is designed to autonomously learn the optimal fuel use from its own historical driving record. It is a fully data-driven and learning-enabled model that does not rely on any prediction or predefined rules. The experiment results show that the proposed model is able to achieve 16.3% energy savings comparing to conventional binary control strategies.

Plug-in hybrid electric vehicles (PHEVs) have been regarded as one of several promising countermeasures to transportation-related energy use and air quality issues. Compared with conventional hybrid electric vehicles, developing an energy management system (EMS) for PHEVs is more challenging due to their more complex powertrain. In this paper, we propose a generic framework of online EMS for PHEVs that is based on an evolutionary algorithm. It includes several control strategies for managing battery state-of-charge (SOC). Extensive simulation testing and evaluation using real-world traffic data indicates that the different SOC control strategies of the proposed online EMS all outperform the conventional control strategy. Out of all the SOC control strategies, the self-adaptive one is the most adaptive to real-time traffic conditions and the most robust to the uncertainties in recharging opportunity. A comparison to the existing models also employing short-term prediction shows that the proposed model can achieve the best fuel economy improvement but requiring less trip information.

The energy management system (EMS) is crucial to a plug-in hybrid electric vehicle (PHEV) in reducing its fuel consumption and pollutant emissions. The EMS determines how energy flows in a hybrid powertrain should be managed in response to a variety of driving conditions. In the development of EMS, the battery state-of-charge (SOC) control strategy plays a critical role. This paper proposes a novel evolutionary algorithm (EA)-based EMS with self-adaptive SOC control strategy for PHEVs, which can achieve the optimal fuel efficiency without trip length (by time) information. Numerical studies show that this proposed system can save up to 13% fuel, compared to other on-line EMS with different SOC control strategies. Further analysis indicates that the proposed system is less sensitive to the errors in predicting propulsion power in real-time, which is favorable for on-line implementation.