Predicting the future motion of traffic agents is crucial for safe and efficient autonomous driving. To this end, we present PredictionNet, a deep neural network (DNN) that predicts the motion of all surrounding traffic agents together with the ego-vehicle's motion. All predictions are probabilistic and are represented in a simple top-down rasterization that allows an arbitrary number of agents. Conditioned on a multi-layer map with lane information, the network outputs future positions, velocities, and backtrace vectors jointly for all agents including the ego-vehicle in a single pass. Trajectories are then extracted from the output. The network can be used to simulate realistic traffic, and it produces competitive results on popular benchmarks. More importantly, it has been used to successfully control a real-world vehicle for hundreds of kilometers, by combining it with a motion planning/control subsystem. The network runs faster than real-time on an embedded GPU, and the system shows good generalization (across sensory modalities and locations) due to the choice of input representation. Furthermore, we demonstrate that by extending the DNN with reinforcement learning (RL), it can better handle rare or unsafe events like aggressive maneuvers and crashes.

Typically, loss functions, regularization mechanisms and other important aspects of training parametric models are chosen heuristically from a limited set of options. In this paper, we take the first step towards automating this process, with the view of producing models which train faster and more robustly. Concretely, we present a meta-learning method for learning parametric loss functions that can generalize across different tasks and model architectures. We develop a pipeline for “meta-training” such loss functions, targeted at maximizing the performance of the model trained under them. The loss landscape produced by our learned losses significantly improves upon the original task-specific losses in both supervised and reinforcement learning tasks. Furthermore, we show that our meta-learning framework is flexible enough to incorporate additional information at meta-train time. This information shapes the learned loss function such that the environment does not need to provide this information during meta-test time. We make our code available at https://sites.google.com/view/mlthree

Quadrotor stabilizing controllers often require careful, model-specific tuning for safe operation. We use reinforcement learning to train policies in simulation that transfer remarkably well to multiple different physical quadrotors. Our policies are low-level, i.e., we map the rotorcrafts' state directly to the motor outputs. The trained control policies are very robust to external disturbances and can withstand harsh initial conditions such as throws. We show how different training methodologies (change of the cost function, modeling of noise, use of domain randomization) might affect flight performance. To the best of our knowledge, this is the first work that demonstrates that a simple neural network can learn a robust stabilizing low-level quadrotor controller (without the use of a stabilizing PD controller) that is shown to generalize to multiple quadrotors. The video of our experiments can be found at https://sites.google.com/view/sim-to-multi-quad.

We present a system to infer and execute a human-readable program from a real-world demonstration. The system consists of a series of neural networks to perform perception, program generation, and program execution. Leveraging convolutional pose machines, the perception network reliably detects the bounding cuboids of objects in real images even when severely occluded, after training only on synthetic images using domain randomization. To increase the applicability of the perception network to new scenarios, the network is formulated to predict in image space rather than in world space. Additional networks detect relationships between objects, generate plans, and determine actions to reproduce a real-world demonstration. The networks are trained entirely in simulation, and the system is tested in the real world on the pick-and-place problem of stacking colored cubes using a Baxter robot.

Manipulation tasks often require robots to make contact between a grasped tool and another object in the robot's environment. The ability to detect and estimate the positions and directions of these contact points is crucial for monitoring the progress of the task, and detecting failures. In this paper, we present a data-driven approach for detecting and localizing contacts between a grasped object and the environment using tactile sensing. We explore framing the contact localization as both a regression and a classification problem and train neural networks accordingly to estimate the contact parameters. We also compare the neural networks with Gaussian process regression and support vector machine classification with spatio-temporal hierarchical matching pursuit feature learning. We evaluate the presented approach using hundreds of contact events on eighteen objects with different shapes, sizes and material properties. The experiments show that the neural network approach can learn to localize contact events for individual objects with a mean absolute error of less than 2.5 cm for the positions and less than 10° for the directions.

We propose a method for controlling multiple active drifters in the presence of external forcing induced by the ocean. Our active drifters have one actuator: they can lower and raise their drogues in depth. By exploiting the vertically stratified nature of ocean currents, we show how classical multi-robot tasks (spreading out and aggregation) can be accomplished by the multi-drifter system. Tests with a realistic simulation based on an ocean model suggest that a practical implementation of active drifters which aggregate and disperse in the coastal ocean could be realized through our control method with relatively inexpensive components. Specifically, we are able to show that over a 90 day deployment a significant fraction of drifters can be made to aggregate in few clusters suitable for recovery.