An accurate model of the environment and the dynamic agents acting in it offers great potential for improving motion planning. We present MILE: a Model-based Imitation LEarning approach to jointly learn a model of the world and a policy for autonomous driving. Our method leverages 3D geometry as an inductive bias and learns a highly compact latent space directly from high-resolution videos of expert demonstrations. Our model is trained on an offline corpus of urban driving data, without any online interaction with the environment. MILE improves upon prior state-of-the-art by 31% in driving score on the CARLA simulator when deployed in a completely new town and new weather conditions. Our model can predict diverse and plausible states and actions, that can be interpretably decoded to bird's-eye view semantic segmentation. Further, we demonstrate that it can execute complex driving manoeuvres from plans entirely predicted in imagination. Our approach is the first camera-only method that models static scene, dynamic scene, and ego-behaviour in an urban driving environment. The code and model weights are available at https://github.com/wayveai/mile.

Hand-crafting generalised decision-making rules for real-world urban autonomous driving is hard. Alternatively, learning behaviour from easy-to-collect human driving demonstrations is appealing. Prior work has studied imitation learning (IL) for autonomous driving with a number of limitations. Examples include only performing lane-following rather than following a user-defined route, only using a single camera view or heavily cropped frames lacking state observability, only lateral (steering) control, but not longitudinal (speed) control and a lack of interaction with traffic. Importantly, the majority of such systems have been primarily evaluated in simulation - a simple domain, which lacks real-world complexities. Motivated by these challenges, we focus on learning representations of semantics, geometry and motion with computer vision for IL from human driving demonstrations. As our main contribution, we present an end-to-end conditional imitation learning approach, combining both lateral and longitudinal control on a real vehicle for following urban routes with simple traffic. We address inherent dataset bias by data balancing, training our final policy on approximately 30 hours of demonstrations gathered over six months. We evaluate our method on an autonomous vehicle by driving 35km of novel routes in European urban streets.

Energy consumption represents one of the most basic constraints for mobile robot autonomy. We propose a new framework to predict energy consumption using information extracted from publicly available maps. This method avoids having to model internal robot configurations, which are often unavailable, while still providing invaluable predictions for both explored and unexplored trajectories. Our approach uses a heteroscedastic Gaussian Process to model the power consumption, which explicitly accounts for variance due to exogenous latent factors such as traffic and weather conditions. We evaluate our framework on 30km of data collected from a city centre environment with a mobile robot travelling on pedestrian walkways. Results across five different test routes show an average difference between predicted and measured power consumption of 3.3%, leading to an average error of 6.6% on predictions of energy consumption. The distinct advantage of our model is our ability to predict measurement variance. The variance predictions improved by 84.3% over a benchmark.

This paper is about building maps which not only contain the traditional information useful for localising — such as point features — but also embeds a spatial model of expected localiser performance. This often overlooked second-order information provides vital context when it comes to map use and planning. Our motivation here is to improve the performance of the popular Teach and Repeat paradigm [1] which has been shown to enable truly large-scale field operation. When using the taught route for localisation, it is often assumed the robot is following exactly, or is sufficiently close to, the original path, enabling successful localisation. However, what happens if it is not possible, or not desirable to exactly follow the mapped path? How far off the beaten track can the robot travel before it gets lost? We present an approach for assessing this localisation area around a taught route, which we refer to as the localisation envelope. Using a combination of physical sampling and a Gaussian Process model, we are able to accurately predict the localisation performance at unseen points.

This paper explores the idea of reducing a robot's energy consumption while following a trajectory by turning off the main localisation subsystem and switching to a lower-powered, less accurate odometry source at appropriate times. This applies to scenarios where the robot is permitted to deviate from the original trajectory, which allows for energy savings. Sensor scheduling is formulated as a probabilistic belief planning problem. Two algorithms are presented which generate feasible perception schedules: the first is based upon a simple heuristic; the second leverages dynamic programming to obtain optimal plans. Both simulations and real-world experiments on a planetary rover prototype demonstrate over 50% savings in perception-related energy, which translates into a 12% reduction in total energy consumption.

In this paper, we consider the problem of Lifelong Robotic Object Discovery (LROD) as the long-term goal of discovering novel objects in the environment while the robot operates, for as long as the robot operates. As a first step towards LROD, we automatically process the raw video stream of an entire workday of a robotic agent to discover objects. We claim that the key to achieve this goal is to incorporate domain knowledge whenever available, in order to detect and adapt to changes in the environment. We propose a general graph-based formulation for LROD in which generic domain knowledge is encoded as constraints. Our formulation enables new sources of domain knowledge—metadata—to be added dynamically to the system, as they become available or as conditions change. By adding domain knowledge, we discover 2.7· more objects and decrease processing time 190 times. Our optimized implementation, HerbDisc, processes 6 h 20 min of RGBD video of real human environments in 18 min 30 s, and discovers 121 correct novel objects with their 3D models.