We introduce RMP-YOLO, a unified framework designed to provide robust motion predictions even with incomplete input data. Our key insight stems from the observation that complete and reliable historical trajectory data plays a pivotal role in ensuring accurate motion prediction. Therefore, we propose a new paradigm that prioritizes the reconstruction of intact historical trajectories before feeding them into the prediction modules. Our approach introduces a novel scene tokenization module to enhance the extraction and fusion of spatial and temporal features. Following this, our proposed recovery module reconstructs agents' incomplete historical trajectories by leveraging local map topology and interactions with nearby agents. The reconstructed, clean historical data is then integrated into the downstream prediction modules. Our framework is able to effectively handle missing data of varying lengths and remains robust against observation noise while maintaining high prediction accuracy. Furthermore, our recovery module is compatible with existing prediction models, ensuring seamless integration. Extensive experiments validate the effectiveness of our approach, and deployment in real-world autonomous vehicles confirms its practical utility. In the 2024 Waymo Motion Prediction Competition, our method, RMP-YOLO, achieves state-of-the-art performance, securing third place. Our code is open-source at https://github.com/ggosjw/RMP-YOLO.

As the field progresses toward Artificial General Intelligence (AGI), there is a pressing need for more comprehensive and insightful evaluation frameworks that go beyond aggregate performance metrics. This paper introduces a unified rating system that jointly models the difficulty of individual test cases and the competency of AI models (or humans) across vision, language, and action domains. Unlike existing metrics that focus solely on models, our approach allows for fine-grained, difficulty-aware evaluations through competitive interactions between models and tasks, capturing both the long-tail distribution of real-world challenges and the competency gap between current models and full task mastery. We validate the generalizability and robustness of our system through extensive experiments on multiple established datasets and models across distinct AGI domains. The resulting rating distributions offer novel perspectives and interpretable insights into task difficulty, model progression, and the outstanding challenges that remain on the path to achieving full AGI task mastery. We have made our code and results publicly available at https://ss47816.github.io/AGI-Elo/.

Adverse weather has raised a big challenge for autonomous vehicles. Unreliable measurements due to sensor degradation could seriously affect the performance of autonomous driving tasks, such as perception and localization. In this work, we study sensor degradation in rainy weather and present a novel method that evaluates the uncertainty for each laser measurement from a 3D LiDAR. With uncertainty estimation, downstream tasks that rely on LiDAR input (e.g., perception or localization) can increase their reliability by adjusting their reliance on laser measurements with varying fidelity. Alternatively, uncertainty estimation can be used for sensor performance evaluation. Our proposed method, SmartRainNet, uses an attention-based Mixture Density Network to model the dependence between neighboring laser measurements and then calculate the probability density for each laser measurement as an uncertainty score. We evaluate SmartRainNet on synthetic and naturalistic sensor degradation datasets and provide qualitative and quantitative results to demonstrate the effectiveness of our method in evaluating uncertainty. Finally, we demonstrate three practical applications of uncertainty estimation to address autonomous driving challenges in rainy weather.

Autonomous driving in rain remains challenging. Rain causes sensor performance degradation that can affect sensor measurement quality. During the rain, lasers may suffer from energy loss due to raindrop absorption. As a result, some laser measurements reflected from obstacles may not be recognized by the LiDAR sensor, thus raising potential risks for autonomous vehicles. This work investigates a novel task that aims to detect those missing measurements. Our solution uses a two-stage learning method to generate an anomaly score for each missing measurement, representing the likelihood of being caused by rain. We evaluate our method with real-world data and demonstrate its effectiveness in identifying anomalous missing measurements through qualitative and quantitative experiments.

Autonomous driving in the rain remains a challenge. One main problem is performance degradation caused by rain. This work introduces a new dataset to study this problem. Our dataset is collected from a full-scale vehicle equipped with a 3D LiDAR sensor and multiple forward-facing cameras under various rainy conditions. In addition, rainfall intensity is recorded in real-time from a rain sensor. The combination of sensor and rainfall intensity measurement is designed for studying algorithm performance under different levels of rainfall. In this work, in addition to presenting dataset creation details, we also introduce three degradation evaluation tasks with baseline results, including rainfall intensity estimation, LiDAR degradation estimation, and 2D object detection evaluation. This dataset, development kit, and baseline codes will be made available at https://smart-rain-dataset.github.io/

Sensor degradation is one of the major challenges for autonomous driving. During the rain, the interference from raindrops can negatively influence LiDAR measurements. For example, valid measurements could be reduced during the rain, and some measurements may become noisy. Unreliable measurements can lead to potential safety issues if autonomous driving systems are unaware of these changes. In this work, we will release a naturalistic driving dataset to advance the research in studying LiDAR degradation. Our dataset consists of 3D LiDAR scans collected by a data collection vehicle under various rainy conditions. Besides these raw scans, we also release LiDAR scan pairs (each pair consists of one scan from rainy weather and one scan from clear weather at the same location). These LiDAR pairs are developed to help researchers identify LiDAR degradation. Finally, we will release a toolbox integrated with mapping, localization, and scan synthesis functions used to create this dataset. This toolbox can facilitate dataset creation for studying degradation in other harsh weather conditions. More information can be found at https://smart-rain-dataset.github.io/.

This paper addresses the safe path planning problem for autonomous mobility with multi-modal perception uncertainties. Specifically, we assume that different sensor inputs lead to different Gaussian process regulated perception uncertainties (named as multi-modal perception uncertainties). We implement a Bayesian inference algorithm, which merges the multi-modal GP-regulated uncertainties into a unified one and translates the unified uncertainty into a dynamic risk map. With the safe path planner taking the risk map as input, we are able to plan a safe path for the autonomous vehicle to follow. Experimental results on an autonomous golf cart testbed validate the applicability and efficiency of the proposed algorithm.

Navigation through dynamic pedestrian environments in a socially compliant manner is still a challenging task for autonomous vehicles. Classical methods usually lead to unnatural vehicle behaviours for pedestrian navigation due to the difficulty in modeling social conventions mathematically. This paper presents an end-to-end path planning system that achieves autonomous navigation in dynamic environments through imitation learning. The proposed system is based on a fully convolutional neural network that maps the raw sensory data into a confidence map for path extraction. Additionally, a classification network is introduced to reduce the unnecessary re-plannings and ensures that the vehicle goes back to the global path when re-planning is not needed. The imitation learning based path planner is implemented on an autonomous wheelchair and tested in a new real-world dynamic pedestrian environment. Experimental results show that the proposed system is able to generate paths for different driving tasks, such as pedestrian following, static and dynamic obstacles avoidance, etc. In comparison to the state-of-the-art method, our system is superior in terms of generating human-like trajectories.

Autonomous driving in rainy conditions remains a big challenge. One of the issues is sensor degradation. LiDAR is commonly used in autonomous driving systems to perceive and understand surrounding environments. However, LiDAR performance can be degraded by rain, thereby influencing other system performance (e.g., perception or localization). Therefore, knowing how much degradation exists in current LiDAR measurements is necessary. Most existing methods can only measure LiDAR degradation in controlled environments (e.g., a chamber with simulated rain); how to quantify LiDAR degradation in dynamic environments while the autonomous vehicle is moving is still a difficult problem. In this work, we propose a novel approach to address this problem using an anomaly detection method. Our method has been evaluated on simulated and real-world data. Experimental results demonstrate the effectiveness of our method to capture LiDAR degradation and yield reasonable degradation estimations. Our experimental data and codes are accessible from http://rain.smart.mit.edu/smartrain/.

Autonomous vehicles on city roads and especially in pedestrian environments require agility to navigate narrow passages and turn in tight spaces, leading to the need for a real-time, robust and adaptable controller. In this paper, we present orientation and context aware controllers for autonomous vehicles that can closely track the reference path wit alh respect to the current state of the vehicle, environmental properties, and the desired target orientation at the desired target location. Our proposed controllers are derived from the widely used pure pursuit controller. We validate our proposed controllers with respect to the baseline pure pursuit controller in simulation and on a full-size autonomous vehicle in a pedestrian environment. Our experimental results suggest significant improvements in adaptability and tracking performance compared to the pure pursuit controller.

This paper investigates the safe path planning problem for an autonomous vehicle operating in unstructured, cluttered environments. While some objects may be accurately with canonical perception algorithms, other objects and clutter may be harder to track. We present an approach that combines two methods of risk assessment: for objects with reliable tracking, we use a Gaussian Process (GP) regulated risk map to describe the risk map information; for unknown objects that we fail to accurately track, we compute a Dynamic Risk Density (DRD) from the overall occupancy and velocity field from LiDAR scan snapshots. Several methods are proposed for combining the GP risk map and DRD, and the resultant hybrid risk map is used for the proposed safe path planning algorithm. Experimental results on an autonomous buggy show that the hybrid risk map is able to yield a safe path planner to navigate the autonomous testbed within the cluttered environments.

Government data identifies driver behaviour errors as a factor in 94% of car crashes, and autonomous vehicles (AVs), which avoids risky driver behaviours completely, are expected to reduce the number of road crashes significantly. Thus, one of the central focuses of developing AVs is to ensure safety during navigation. However, in reality, AV safety has been far below its expectation, and so far, no government has allowed for complete autonomous driving without human supervision. This paper proposes a dynamic safe path planning algorithm for AVs with Gaussian process regulated risk map. By reasonably assuming that the output of the object detection and tracking module follows a multi-variate Gaussian distribution, we put forward a safe path planning paradigm with Gaussian process regulated risk map, ensuring safety with high confidence. Both simulation results and in-vehicle tests demonstrate the effectiveness of the proposed algorithm.