Demo Data from EDBT 2026 / reviewers in the wild / expert
Alex Irpan
dblp:202/2063 · also Alexander Irpan
· DBLP profile ↗
13ranked-venue papers
1as first author
4since 2021 · last 2025
—ORCID · none
Domains — the database's venue-derived domains; a paper can count in several
Artificial intelligence and machine learning · 13 · 1 first-author · 4 since 2021Systems, architecture and hardware · 4 · 2 since 2021Graphics, computer vision, multimedia, augmented reality and games · 2
Expertise — from the expertise taxonomy: the topics of the expert's papers under the CCF categories. A weight counts papers with recency: 1 for a paper about the topic, 0.3 when the topic is its context, halved every five years.
| Artificial intelligence
12 papers |
Reinforcement learning · 34% Motion planning and robot control · 21% Transfer learning and domain adaptation · 18% | |
| Theoretical computer science
1 paper |
Combinatorics and discrete mathematics · 100% |
Topics — the 30 heaviest of 33, each with the papers that count most for it
| Topic | Weight | Papers | Last | Evidence papers |
|---|---|---|---|---|
Robotics › Motion planning and robot control
robot learning |
1.7 | 3 | 2024 | Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration · ICRA 2024 Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills · ICML 2021 Scalable Multi-Task Imitation Learning with Autonomous Improvement · ICRA 2020 |
Machine learning › Transfer learning and domain adaptation
sim-to-real transfer |
0.9 | 3 | 2020 | RL-CycleGAN: Reinforcement Learning Aware Simulation-to-Real · CVPR 2020 Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks · CVPR 2019 Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping · ICRA 2018 |
Machine learning › Reinforcement learning
value-based reinforcement learning |
0.9 | 2 | 2024 | Stop Regressing: Training Value Functions via Classification for Scalable Deep RL · ICML 2024 Off-Policy Evaluation via Off-Policy Classification · NeurIPS 2019 |
Robotics › Motion planning and robot control › robot learning › manipulation learning
language-conditioned manipulation |
0.9 | 1 | 2025 | STEER: Flexible Robotic Manipulation via Dense Language Grounding · ICRA 2025 |
Computer vision › Vision and language › visual grounding
language grounding |
0.9 | 1 | 2025 | STEER: Flexible Robotic Manipulation via Dense Language Grounding · ICRA 2025 |
Robotics › Robot manipulation
grasping |
0.8 | 3 | 2020 | Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks · CVPR 2019 Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping · ICRA 2018 RL-CycleGAN: Reinforcement Learning Aware Simulation-to-Real · CVPR 2020 |
Machine learning › Deep learning architectures and training › loss function design
classification loss function |
0.8 | 1 | 2024 | Stop Regressing: Training Value Functions via Classification for Scalable Deep RL · ICML 2024 |
Robotics › Motion planning and robot control › robot learning › robot policy learning
generalist robot policy |
0.8 | 1 | 2024 | Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration · ICRA 2024 |
Machine learning › Reinforcement learning
value function estimation |
0.8 | 1 | 2024 | Stop Regressing: Training Value Functions via Classification for Scalable Deep RL · ICML 2024 |
Machine learning › Transfer learning and domain adaptation
domain adaptation |
0.7 | 2 | 2019 | Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks · CVPR 2019 Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping · ICRA 2018 |
Machine learning › Reinforcement learning
goal-conditioned reinforcement learning |
0.5 | 1 | 2021 | Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills · ICML 2021 |
Machine learning › Reinforcement learning
offline reinforcement learning |
0.5 | 1 | 2021 | Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills · ICML 2021 |
Machine learning › Generative modeling › generative adversarial network
image-to-image translation |
0.4 | 1 | 2020 | RL-CycleGAN: Reinforcement Learning Aware Simulation-to-Real · CVPR 2020 |
Machine learning › Reinforcement learning
imitation learning |
0.4 | 1 | 2020 | Scalable Multi-Task Imitation Learning with Autonomous Improvement · ICRA 2020 |
Machine learning › Transfer learning and domain adaptation
meta-learning |
0.4 | 1 | 2020 | Meta-Learning Requires Meta-Augmentation · NeurIPS 2020 |
Machine learning › Reinforcement learning › imitation learning › transfer imitation learning
multi-task imitation learning |
0.4 | 1 | 2020 | Scalable Multi-Task Imitation Learning with Autonomous Improvement · ICRA 2020 |
Machine learning › Transfer learning and domain adaptation › sim-to-real transfer
domain randomization |
0.4 | 1 | 2019 | Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks · CVPR 2019 |
Machine learning › Learning theory
model selection |
0.4 | 1 | 2019 | Off-Policy Evaluation via Off-Policy Classification · NeurIPS 2019 |
Machine learning › Reinforcement learning
off-policy evaluation |
0.4 | 1 | 2019 | Off-Policy Evaluation via Off-Policy Classification · NeurIPS 2019 |
Machine learning › Reinforcement learning
deep reinforcement learning |
0.3 | 1 | 2018 | Can Deep Reinforcement Learning Solve Erdos-Selfridge-Spencer Games? · ICML 2018 |
Machine learning › Reinforcement learning
generalization in reinforcement learning |
0.3 | 1 | 2018 | Can Deep Reinforcement Learning Solve Erdos-Selfridge-Spencer Games? · ICML 2018 |
Machine learning › Reinforcement learning
multi-agent reinforcement learning |
0.3 | 1 | 2018 | Can Deep Reinforcement Learning Solve Erdos-Selfridge-Spencer Games? · ICML 2018 |
Machine learning › Transfer learning and domain adaptation › domain adaptation › visual domain adaptation
pixel-level domain adaptation |
0.3 | 1 | 2018 | Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping · ICRA 2018 |
Machine learning › Reinforcement learning › multi-agent reinforcement learning
self-play |
0.3 | 1 | 2018 | Can Deep Reinforcement Learning Solve Erdos-Selfridge-Spencer Games? · ICML 2018 |
Machine learning › Deep learning architectures and training
recurrent neural network |
0.3 | 1 | 2017 | Learning Hierarchical Information Flow with Recurrent Neural Modules · NIPS 2017 |
Machine learning › Learning theory › generalization
generalization theory |
0.1 | 1 | 2020 | Meta-Learning Requires Meta-Augmentation · NeurIPS 2020 |
Machine learning › Learning paradigms › weakly supervised learning
positive-unlabeled learning |
0.1 | 1 | 2019 | Off-Policy Evaluation via Off-Policy Classification · NeurIPS 2019 |
Machine learning › Reinforcement learning › value-based reinforcement learning
q-value estimation |
0.1 | 1 | 2019 | Off-Policy Evaluation via Off-Policy Classification · NeurIPS 2019 |
Combinatorics and discrete mathematics
combinatorial game |
0.1 | 1 | 2018 | Can Deep Reinforcement Learning Solve Erdos-Selfridge-Spencer Games? · ICML 2018 |
Machine learning › Representation and self-supervised learning › hierarchical representation › hierarchical representation learning
hierarchical feature learning |
0.1 | 1 | 2017 | Learning Hierarchical Information Flow with Recurrent Neural Modules · NIPS 2017 |
Methods — techniques the papers use, named apart from their topics
vision-language model · 0.9policy training · 0.9dense language annotation · 0.9reinforcement learning · 0.8generative adversarial network · 0.8transformer policy · 0.8mean squared error regression · 0.8large-scale pretraining · 0.8categorical cross entropy · 0.8goal chaining · 0.5deep reinforcement learning · 0.3
| Year | Publication | Venue | Position |
|---|---|---|---|
| 2025 | STEER: Flexible Robotic Manipulation via Dense Language GroundingabstractThe complexity of the real world demands robotic systems that can intelligently adapt to unseen situations. We present STEER, a robot learning framework that bridges highlevel, commonsense reasoning with precise, flexible low-level control. Our approach translates complex situational awareness into actionable low-level behavior through training languagegrounded policies with dense annotation. By structuring policy training around fundamental, modular manipulation skills expressed in natural language, STEER exposes an expressive interface for humans or Vision-Language Models (VLMs) to intelligently orchestrate the robot's behavior by reasoning about the task and context. Our experiments demonstrate the skills learned via STEER can be combined to synthesize novel behaviors to adapt to new situations or perform completely new tasks without additional data collection or training. Project website: https://lauramsmith.github.io/steer Laura Smith 0001, Alex Irpan, Montse Gonzalez Arenas, Sean Kirmani, Dmitry Kalashnikov, Dhruv Shah, Ted Xiao |
ICRA | 2 |
| 2024 | Stop Regressing: Training Value Functions via Classification for Scalable Deep RLabstractValue functions are an essential component in deep reinforcement learning (RL), that are typically trained via mean squared error regression to match bootstrapped target values. However, scaling value-based RL methods to large networks has proven challenging. This difficulty is in stark contrast to supervised learning: by leveraging a cross-entropy classification loss, supervised methods have scaled reliably to massive networks. Observing this discrepancy, in this paper, we investigate whether the scalability of deep RL can also be improved simply by using classification in place of regression for training value functions. We show that training value functions with categorical cross-entropy significantly enhances performance and scalability across various domains, including single-task RL on Atari 2600 games, multi-task RL on Atari with large-scale ResNets, robotic manipulation with Q-transformers, playing Chess without search, and a language-agent Wordle task with high-capacity Transformers, achieving state-of-the-art results on these domains. Through careful analysis, we show that categorical cross-entropy mitigates issues inherent to value-based RL, such as noisy targets and non-stationarity. We argue that shifting to categorical cross-entropy for training value functions can substantially improve the scalability of deep RL at little-to-no cost. Jesse Farebrother, Jordi Orbay, Adrien Ali Taïga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, Rishabh Agarwal |
ICML | 7 |
| 2024 | Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment CollaborationabstractLarge, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train "generalist" X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. The project website is robotics-transformer-x.github.io. Abigail O'Neill, Abhiram Maddukuri, Abhishek Gupta 0004, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew E. Wang, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu 0003, Charlotte Le, Chelsea Finn, Chen Wang 0053, Chenfeng Xu, Cheng Chi 0001, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Drieß, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Paul Foster, Fangchen Liu, Federico Ceola, Fei Xia 0002, Feiyu Zhao, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guanzhi Wang, Hao Su 0001, Haoshu Fang, Henghui Bao, Heni Ben Amor, Henrik I. Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim 0001, Jaimyn Drake, Jan Peters 0001, Jan Schneider 0007, Jasmine Hsu, Jeannette Bohg, Jeffrey T. Bingham, Jensen Gao, Jiaheng Hu, Jiajun Wu 0001, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan 0001, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Kenneth Y. Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Zhang 0002, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Yunliang Chen 0001, Lerrel Pinto, Li Fei-Fei 0001, Liam Tan, Linxi Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang 0003, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma 0001, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen 0001, Nicolas Heess, Nikhil J. Joshi, Niko Sünderhauf, Norman Di Palo, Nur Muhammad Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R. Sanketi, Patrick Tree Miller, Patrick Yin, Paul Wohlhart, Peng Xu 0010, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Rafael Rafailov, Ria Doshi, Roberto Martin Martin, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante-Gomez, Sean Kirmani, Sergey Levine, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham D. Sonawani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair 0003, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vincent Vanhoucke, Wolfram Burgard, Xiaolong Wang 0004, Xinghao Zhu, Xinyang Geng, Liangwei Xu, Yecheng Jason Ma 0001, Yejin Kim 0003, Yevgen Chebotar, Yilin Wu 0003, Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang 0001, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zichen Jeff Cui, Zichen Zhang 0016, Zipeng Lin |
ICRA | 14 |
| 2021 | Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic SkillsabstractWe consider the problem of learning useful robotic skills from previously collected offline data without access to manually specified rewards or additional online exploration, a setting that is becoming increasingly important for scaling robot learning by reusing past robotic data. In particular, we propose the objective of learning a functional understanding of the environment by learning to reach any goal state in a given dataset. We employ goal-conditioned Q-learning with hindsight relabeling and develop several techniques that enable training in a particularly challenging offline setting. We find that our method can operate on high-dimensional camera images and learn a variety of skills on real robots that generalize to previously unseen scenes and objects. We also show that our method can learn to reach long-horizon goals across multiple episodes through goal chaining, and learn rich representations that can help with downstream tasks through pre-training or auxiliary objectives. Yevgen Chebotar, Karol Hausman, Yao Lu 0006, Ted Xiao, Dmitry Kalashnikov, Jacob Varley, Alex Irpan, Benjamin Eysenbach, Ryan Julian, Chelsea Finn, Sergey Levine |
ICML | 7 |
| 2020 | RL-CycleGAN: Reinforcement Learning Aware Simulation-to-RealabstractDeep neural network based reinforcement learning (RL) can learn appropriate visual representations for complex tasks like vision-based robotic grasping without the need for manually engineering or prior learning a perception system. However, data for RL is collected via running an agent in the desired environment, and for applications like robotics, running a robot in the real world may be extremely costly and time consuming. Simulated training offers an appealing alternative, but ensuring that policies trained in simulation can transfer effectively into the real world requires additional machinery. Simulations may not match reality, and typically bridging the simulation-to-reality gap requires domain knowledge and task-specific engineering. We can automate this process by employing generative models to translate simulated images into realistic ones. However, this sort of translation is typically task-agnostic, in that the translated images may not preserve all features that are relevant to the task. In this paper, we introduce the RL-scene consistency loss for image translation, which ensures that the translation operation is invariant with respect to the Q-values associated with the image. This allows us to learn a task-aware translation. Incorporating this loss into unsupervised domain translation, we obtain the RL-CycleGAN, a new approach for simulation-to-real-world transfer for reinforcement learning. In evaluations of RL-CycleGAN on two vision-based robotics grasping tasks, we show that RL-CycleGAN offers a substantial improvement over a number of prior methods for sim-to-real transfer, attaining excellent real-world performance with only a modest number of real-world observations. Kanishka Rao, Chris Harris 0001, Alex Irpan, Sergey Levine, Julian Ibarz, Mohi Khansari |
CVPR | 3 |
| 2020 | Scalable Multi-Task Imitation Learning with Autonomous ImprovementabstractWhile robot learning has demonstrated promising results for enabling robots to automatically acquire new skills, a critical challenge in deploying learning-based systems is scale: acquiring enough data for the robot to effectively generalize broadly. Imitation learning, in particular, has remained a stable and powerful approach for robot learning, but critically relies on expert operators for data collection. In this work, we target this challenge, aiming to build an imitation learning system that can continuously improve through autonomous data collection, while simultaneously avoiding the explicit use of reinforcement learning, to maintain the stability, simplicity, and scalability of supervised imitation. To accomplish this, we cast the problem of imitation with autonomous improvement into a multi-task setting. We utilize the insight that, in a multi-task setting, a failed attempt at one task might represent a successful attempt at another task. This allows us to leverage the robot's own trials as demonstrations for tasks other than the one that the robot actually attempted. Using an initial dataset of multitask demonstration data, the robot autonomously collects trials which are only sparsely labeled with a binary indication of whether the trial accomplished any useful task or not. We then embed the trials into a learned latent space of tasks, trained using only the initial demonstration dataset, to draw similarities between various trials, enabling the robot to achieve one-shot generalization to new tasks. In contrast to prior imitation learning approaches, our method can autonomously collect data with sparse supervision for continuous improvement, and in contrast to reinforcement learning algorithms, our method can effectively improve from sparse, task-agnostic reward signals. Avi Singh, Eric Jang, Alex Irpan, Daniel Kappler, Murtaza Dalal, Sergey Levine, Mohi Khansari, Chelsea Finn |
ICRA | 3 |
| 2020 | Meta-Learning Requires Meta-AugmentationabstractMeta-learning algorithms aim to learn two components: a model that predicts targets for a task, and a base learner that updates that model when given examples from a new task. This additional level of learning can be powerful, but it also creates another potential source of overfitting, since we can now overfit in either the model or the base learner. We describe both of these forms of meta-learning overfitting, and demonstrate that they appear experimentally in common meta-learning benchmarks. We introduce an information-theoretic framework of meta-augmentation, whereby adding randomness discourages the base learner and model from learning trivial solutions that do not generalize to new tasks. We demonstrate that meta-augmentation produces large complementary benefits to recently proposed meta-regularization techniques. Janarthanan Rajendran, Alex Irpan, Eric Jang |
NeurIPS | 2 |
| 2019 | Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation NetworksabstractReal world data, especially in the domain of robotics, is notoriously costly to collect. One way to circumvent this can be to leverage the power of simulation to produce large amounts of labelled data. However, training models on simulated images does not readily transfer to real-world ones. Using domain adaptation methods to cross this "reality gap" requires a large amount of unlabelled real-world data, whilst domain randomization alone can waste modeling power. In this paper, we present Randomized-to-Canonical Adaptation Networks (RCANs), a novel approach to crossing the visual reality gap that uses no real-world data. Our method learns to translate randomized rendered images into their equivalent non-randomized, canonical versions. This in turn allows for real images to also be translated into canonical sim images. We demonstrate the effectiveness of this sim-to-real approach by training a vision-based closed-loop grasping reinforcement learning agent in simulation, and then transferring it to the real world to attain 70% zero-shot grasp success on unseen objects, a result that almost doubles the success of learning the same task directly on domain randomization alone. Additionally, by joint finetuning in the real-world with only 5,000 real-world grasps, our method achieves 91%, attaining comparable performance to a state-of-the-art system trained with 580,000 real-world grasps, resulting in a reduction of real-world data by more than 99%. Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, Konstantinos Bousmalis |
CVPR | 5 |
| 2019 | Off-Policy Evaluation via Off-Policy ClassificationabstractIn this work, we consider the problem of model selection for deep reinforcement learning (RL) in real-world environments. Typically, the performance of deep RL algorithms is evaluated via on-policy interactions with the target environment. However, comparing models in a real-world environment for the purposes of early stopping or hyperparameter tuning is costly and often practically infeasible. This leads us to examine off-policy policy evaluation (OPE) in such settings. We focus on OPE of value-based methods, which are of particular interest in deep RL with applications like robotics, where off-policy algorithms based on Q-function estimation can often attain better sample complexity than direct policy optimization. Furthermore, existing OPE metrics either rely on a model of the environment, or the use of importance sampling (IS) to correct for the data being off-policy. However, for high-dimensional observations, such as images, models of the environment can be difficult to fit and value-based methods can make IS hard to use or even ill-conditioned, especially when dealing with continuous action spaces. In this paper, we focus on the specific case of MDPs with continuous action spaces and sparse binary rewards, which is representative of many important real-world applications. We propose an alternative metric that relies on neither models nor IS, by framing OPE as a positive-unlabeled (PU) classification problem. We experimentally show that this metric outperforms baselines on a number of tasks. Most importantly, it can reliably predict the relative performance of different policies in a number of generalization scenarios, including the transfer to the real-world of policies trained in simulation for an image-based robotic manipulation task. Alex Irpan, Kanishka Rao, Konstantinos Bousmalis, Chris Harris 0001, Julian Ibarz, Sergey Levine |
NeurIPS | 1 |
| 2019 | Noise Contrastive Priors for Functional Uncertainty
Danijar Hafner, Dustin Tran, Timothy P. Lillicrap, Alex Irpan, James Davidson |
UAI | 4 |
| 2018 | Can Deep Reinforcement Learning Solve Erdos-Selfridge-Spencer Games?abstractDeep reinforcement learning has achieved many recent successes, but our understanding of its strengths and limitations is hampered by the lack of rich environments in which we can fully characterize optimal behavior, and correspondingly diagnose individual actions against such a characterization. Here we consider a family of combinatorial games, arising from work of Erdos, Selfridge, and Spencer, and we propose their use as environments for evaluating and comparing different approaches to reinforcement learning. These games have a number of appealing features: they are challenging for current learning approaches, but they form (i) a low-dimensional, simply parametrized environment where (ii) there is a linear closed form solution for optimal behavior from any state, and (iii) the difficulty of the game can be tuned by changing environment parameters in an interpretable way. We use these Erdos-Selfridge-Spencer games not only to compare different algorithms, but test for generalization, make comparisons to supervised learning, analyse multiagent play, and even develop a self play algorithm. Maithra Raghu, Alex Irpan, Jacob Andreas, Robert D. Kleinberg, Quoc V. Le, Jon M. Kleinberg |
ICML | 2 |
| 2018 | Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic GraspingabstractInstrumenting and collecting annotated visual grasping datasets to train modern machine learning algorithms can be extremely time-consuming and expensive. An appealing alternative is to use off-the-shelf simulators to render synthetic data for which ground-truth annotations are generated automatically. Unfortunately, models trained purely on simulated data often fail to generalize to the real world. We study how randomized simulated environments and domain adaptation methods can be extended to train a grasping system to grasp novel objects from raw monocular RGB images. We extensively evaluate our approaches with a total of more than 25,000 physical test grasps, studying a range of simulation conditions and domain adaptation methods, including a novel extension of pixel-level domain adaptation that we term the GraspGAN. We show that, by using synthetic data and domain adaptation, we are able to reduce the number of real-world samples needed to achieve a given level of performance by up to 50 times, using only randomly generated simulated objects. We also show that by using only unlabeled real-world data and our GraspGAN methodology, we obtain real-world grasping performance without any real-world labels that is similar to that achieved with 939,777 labeled real-world samples. Konstantinos Bousmalis, Alex Irpan, Paul Wohlhart, Matthew Kelcey, Mrinal Kalakrishnan, Laura Downs, Julian Ibarz, Peter Pastor, Kurt Konolige, Sergey Levine, Vincent Vanhoucke |
ICRA | 2 |
| 2017 | Learning Hierarchical Information Flow with Recurrent Neural ModulesabstractWe propose ThalNet, a deep learning model inspired by neocortical communication via the thalamus. Our model consists of recurrent neural modules that send features through a routing center, endowing the modules with the flexibility to share features over multiple time steps. We show that our model learns to route information hierarchically, processing input data by a chain of modules. We observe common architectures, such as feed forward neural networks and skip connections, emerging as special cases of our architecture, while novel connectivity patterns are learned for the text8 compression task. Our model outperforms standard recurrent neural networks on several sequential benchmarks. Danijar Hafner, Alex Irpan, James Davidson, Nicolas Heess |
NIPS | 2 |