State of the art reinforcement learning has enabled training agents on tasks of ever increasing complexity. However, the current paradigm tends to favor training agents from scratch on every new task or on collections of tasks with a view towards generalizing to novel task configurations. The former suffers from poor data efficiency while the latter is difficult when test tasks are out-of-distribution. Agents that can effectively transfer their knowledge about the world pose a potential solution to these issues. In this paper, we investigate transfer learning in the context of model-based agents. Specifically, we aim to understand where exactly environment models have an advantage and why. We find that a model-based approach outperforms controlled model-free baselines for transfer learning. Through ablations, we show that both the policy and dynamics model learnt through exploration matter for successful transfer. We demonstrate our results across three domains which vary in their requirements for transfer: in-distribution procedural (Crafter), in-distribution identical (RoboDesk), and out-of-distribution (Meta-World). Our results show that intrinsic exploration combined with environment models present a viable direction towards agents that are self-supervised and able to generalize to novel reward functions.

Designing physical artifacts that serve a purpose---such as tools and other functional structures---is central to engineering as well as everyday human behavior. Though automating design using machine learning has tremendous promise, existing methods are often limited by the task-dependent distributions they were exposed to during training. Here we showcase a task-agnostic approach to inverse design, by combining general-purpose graph network simulators with gradient-based design optimization. This constitutes a simple, fast, and reusable approach that solves high-dimensional problems with complex physical dynamics, including designing surfaces and tools to manipulate fluid flows and optimizing the shape of an airfoil to minimize drag. This framework produces high-quality designs by propagating gradients through trajectories of hundreds of steps, even when using models that were pre-trained for single-step predictions on data substantially different from the design tasks. In our fluid manipulation tasks, the resulting designs outperformed those found by sampling-based optimization techniques. In airfoil design, they matched the quality of those obtained with a specialized solver. Our results suggest that despite some remaining challenges, machine learning-based simulators are maturing to the point where they can support general-purpose design optimization across a variety of fluid-structure interaction domains.

Physical construction—the ability to compose objects, subject to physical dynamics, to serve some function—is fundamental to human intelligence. We introduce a suite of challenging physical construction tasks inspired by how children play with blocks, such as matching a target configuration, stacking blocks to connect objects together, and creating shelter-like structures over target objects. We examine how a range of deep reinforcement learning agents fare on these challenges, and introduce several new approaches which provide superior performance. Our results show that agents which use structured representations (e.g., objects and scene graphs) and structured policies (e.g., object-centric actions) outperform those which use less structured representations, and generalize better beyond their training when asked to reason about larger scenes. Model-based agents which use Monte-Carlo Tree Search also outperform strictly model-free agents in our most challenging construction problems. We conclude that approaches which combine structured representations and reasoning with powerful learning are a key path toward agents that possess rich intuitive physics, scene understanding, and planning.

Many courses in scientific fields require students not just to write code, but to also visualize their data, work through a mathematical derivation, or write a paragraph interpreting their results. The Jupyter notebook (formerly known as the IPython notebook) is an ideal platform for creating assignments that include all of these question types, due to its interactive document format that weaves together code, prose, images, and math. Instructors can provide students with self-contained notebooks that include instructions, coding exercises, and written responses all in the same place. Students can write and execute code in these notebooks, and any text or images produced by the code are displayed immediately beneath the input that produced them. However, while having all exercises in the same place makes for better assignments, it also makes them difficult to grade. This demo presents nbgrader, an open-source tool developed by the Jupyter project that solves this problem by providing an interface that blends the autograding of notebook-based assignments with manual human grading. Additionally, nbgrader streamlines the process of assignment creation, distribution, collection, grading, and feedback-a process that is often logistically difficult for instructors. To give students access to the notebook, this demo illustrates how it can be deployed in the cloud, and how this complements the grading process. Information about the Jupyter notebook can be found at http://jupyter.org/, and a demo of the notebook at https://try.jupyter.org/. Information about nbgrader can be found at https://github.com/jupyter/nbgrader. A laptop is recommended for this demo. Handouts will be provided.