Demo Data from VLDB 2026 / reviewers in the wild / expert
Siddharth Karamcheti
dblp:199/1922
· DBLP profile ↗
11ranked-venue papers
2as first author
8since 2021 · last 2024
0000-0003-2153-2455ORCID · corroborated
Domains — the database's venue-derived domains; a paper can count in several
Artificial intelligence and machine learning · 11 · 2 first-author · 8 since 2021Systems, architecture and hardware · 2 · 2 since 2021Graphics, computer vision, multimedia, augmented reality and games · 1Human-computer interaction and ubiquitous computing · 1 · 1 since 2021
Expertise — from the expertise taxonomy: the topics of the expert's papers under the CCF categories. A weight counts papers with recency: 1 for a paper about the topic, 0.3 when the topic is its context, halved every five years.
| Artificial intelligence
11 papers |
Reinforcement learning · 23% Vision and language · 20% Motion planning and robot control · 13% | |
| Human-computer interaction and pervasive computing
2 papers |
Human-robot interaction · 60% Games and playful interaction · 40% |
Topics — the 27 heaviest of 32, each with the papers that count most for it
| Topic | Weight | Papers | Last | Evidence papers |
|---|---|---|---|---|
Computer vision › Vision and language
visual question answering |
1.3 | 2 | 2024 | Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models · ICML 2024 Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering · ACL/IJCNLP (1) 2021 |
Knowledge, reasoning and agents › Knowledge representation and reasoning
commonsense reasoning |
0.9 | 2 | 2024 | Toward Grounded Commonsense Reasoning · ICRA 2024 Generating Interactive Worlds with Text · AAAI 2020 |
Robotics › Robot navigation and mapping
active perception |
0.8 | 1 | 2024 | Toward Grounded Commonsense Reasoning · ICRA 2024 |
Robotics › Motion planning and robot control › robot learning › robot policy learning
generalist robot policy |
0.8 | 1 | 2024 | Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration · ICRA 2024 |
Robotics › Motion planning and robot control
robot learning |
0.8 | 1 | 2024 | Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration · ICRA 2024 |
Knowledge, reasoning and agents › Planning, search and constraint satisfaction
task planning |
0.8 | 1 | 2024 | Toward Grounded Commonsense Reasoning · ICRA 2024 |
Computer vision › Vision and language
multimodal training data |
0.7 | 1 | 2023 | OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents · NeurIPS 2023 |
Human-robot interaction
shared control |
0.7 | 1 | 2023 | No, to the Right: Online Language Corrections for Robotic Manipulation via Shared Autonomy · HRI 2023 |
Machine learning › Efficient and distributed learning
active learning |
0.5 | 1 | 2021 | Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering · ACL/IJCNLP (1) 2021 |
Knowledge, reasoning and agents › Multi-agent systems
automated negotiation |
0.5 | 1 | 2021 | Targeted Data Acquisition for Evolving Negotiation Agents · ICML 2021 |
Machine learning › Reinforcement learning
exploration |
0.5 | 1 | 2021 | ELLA: Exploration through Learned Language Abstraction · NeurIPS 2021 |
Machine learning › Reinforcement learning › reward design › reward shaping
language-based reward shaping |
0.5 | 1 | 2021 | ELLA: Exploration through Learned Language Abstraction · NeurIPS 2021 |
Machine learning › Reinforcement learning
multi-agent reinforcement learning |
0.5 | 1 | 2021 | Targeted Data Acquisition for Evolving Negotiation Agents · ICML 2021 |
Machine learning › Reinforcement learning › reward design
reward shaping |
0.5 | 1 | 2021 | ELLA: Exploration through Learned Language Abstraction · NeurIPS 2021 |
Machine learning › Reinforcement learning › exploration › efficient exploration
sample-efficient exploration |
0.5 | 1 | 2021 | ELLA: Exploration through Learned Language Abstraction · NeurIPS 2021 |
Machine learning › Reinforcement learning › sample efficiency
sample-efficient reinforcement learning |
0.5 | 1 | 2021 | Targeted Data Acquisition for Evolving Negotiation Agents · ICML 2021 |
Games and playful interaction › game AI
procedural content generation |
0.4 | 1 | 2020 | Generating Interactive Worlds with Text · AAAI 2020 |
Natural language and speech › Question answering and dialogue systems
dialogue |
0.4 | 1 | 2019 | Learning to Speak and Act in a Fantasy Text Adventure Game · EMNLP/IJCNLP (1) 2019 |
Natural language and speech › Question answering and dialogue systems
evidence selection |
0.4 | 1 | 2019 | Finding Generalizable Evidence by Learning to Convince Q&A Models · EMNLP/IJCNLP (1) 2019 |
Machine learning › Trustworthy machine learning
hallucination |
0.2 | 1 | 2024 | Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models · ICML 2024 |
Machine learning › Trustworthy machine learning
interpretability |
0.2 | 1 | 2024 | Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models · ICML 2024 |
Computer vision › Image recognition and object detection
object localization |
0.2 | 1 | 2024 | Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models · ICML 2024 |
Robotics › Motion planning and robot control › robot learning › manipulation learning
language-conditioned manipulation |
0.2 | 1 | 2023 | No, to the Right: Online Language Corrections for Robotic Manipulation via Shared Autonomy · HRI 2023 |
Machine learning › Reinforcement learning
imitation learning |
0.1 | 1 | 2021 | Targeted Data Acquisition for Evolving Negotiation Agents · ICML 2021 |
Natural language and speech › Language models and text generation
instruction following |
0.1 | 1 | 2021 | ELLA: Exploration through Learned Language Abstraction · NeurIPS 2021 |
Natural language and speech › Language models and text generation
natural language instructions |
0.1 | 1 | 2021 | ELLA: Exploration through Learned Language Abstraction · NeurIPS 2021 |
Natural language and speech › Question answering and dialogue systems › reasoning-based question answering
multi-hop question answering |
0.1 | 1 | 2019 | Finding Generalizable Evidence by Learning to Convince Q&A Models · EMNLP/IJCNLP (1) 2019 |
Methods — techniques the papers use, named apart from their topics
imitation learning · 1.7user study · 1.3latent actions · 1.3reinforcement learning · 0.9visual representation pretraining · 0.8vision-language model · 0.8transformer policy · 0.8large-scale pretraining · 0.8large language model · 0.8instruction tuning · 0.8neural network · 0.4compositional generation · 0.4
| Year | Publication | Venue | Position |
|---|---|---|---|
| 2024 | Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language ModelsabstractVisually-conditioned language models (VLMs) have seen growing adoption in applications such as visual dialogue, scene understanding, and robotic task planning; adoption that has fueled a wealth of new models such as LLaVa, InstructBLIP, and PaLI-3. Despite the volume of new releases, key design decisions around image preprocessing, architecture, and optimization are under-explored, making it challenging to understand what factors account for model performance – a challenge further complicated by the lack of objective, consistent evaluations. To address these gaps, we first compile a suite of standardized evaluations spanning visual question answering, object localization, and challenge sets that probe properties such as hallucination; evaluations that provide fine-grained insight VLM capabilities. Second, we rigorously investigate VLMs along key design axes, including pretrained visual representations and training from base vs. instruct-tuned language models, amongst others. We couple our analysis with three resource contributions: (1) a unified framework for evaluating VLMs, (2) optimized, flexible training code, and (3) checkpoints for all models, including a family of VLMs at the 7-13B scale that strictly outperform InstructBLIP and LLaVa v1.5, the state-of-the-art in open VLMs. Siddharth Karamcheti, Suraj Nair 0003, Ashwin Balakrishna, Percy Liang, Thomas Kollar, Dorsa Sadigh |
ICML | 1 |
| 2024 | Toward Grounded Commonsense ReasoningabstractConsider a robot tasked with tidying a desk with a meticulously constructed Lego sports car. A human may recognize that it is not appropriate to disassemble the sports car and put it away as part of the "tidying." How can a robot reach that conclusion? Although large language models (LLMs) have recently been used to enable commonsense reasoning, grounding this reasoning in the real world has been challenging. To reason in the real world, robots must go beyond passively querying LLMs and actively gather information from the environment that is required to make the right decision. For instance, after detecting that there is an occluded car, the robot may need to actively perceive the car to know whether it is an advanced model car made out of Legos or a toy car built by a toddler. We propose an approach that leverages an LLM and vision language model (VLM) to help a robot actively perceive its environment to perform grounded commonsense reasoning. To evaluate our framework at scale, we release the MessySurfaces dataset which contains images of 70 real-world surfaces that need to be cleaned. We additionally illustrate our approach with a robot on 2 carefully designed surfaces. We find an average 12.9% improvement on the MessySurfaces benchmark and an average 15% improvement on the robot experiments over baselines that do not use active perception. The dataset, code, and videos of our approach can be found at https://minaek.github.io/grounded_commonsense_reasoning/. Minae Kwon, Hengyuan Hu, Vivek Myers, Siddharth Karamcheti, Anca D. Dragan, Dorsa Sadigh |
ICRA | 4 |
| 2024 | Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment CollaborationabstractLarge, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train "generalist" X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. The project website is robotics-transformer-x.github.io. Abigail O'Neill, Abhiram Maddukuri, Abhishek Gupta 0004, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew E. Wang, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu 0003, Charlotte Le, Chelsea Finn, Chen Wang 0053, Chenfeng Xu, Cheng Chi 0001, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Drieß, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Paul Foster, Fangchen Liu, Federico Ceola, Fei Xia 0002, Feiyu Zhao, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guanzhi Wang, Hao Su 0001, Haoshu Fang, Henghui Bao, Heni Ben Amor, Henrik I. Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim 0001, Jaimyn Drake, Jan Peters 0001, Jan Schneider 0007, Jasmine Hsu, Jeannette Bohg, Jeffrey T. Bingham, Jensen Gao, Jiaheng Hu, Jiajun Wu 0001, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan 0001, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Kenneth Y. Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Zhang 0002, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Yunliang Chen 0001, Lerrel Pinto, Li Fei-Fei 0001, Liam Tan, Linxi Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang 0003, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma 0001, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen 0001, Nicolas Heess, Nikhil J. Joshi, Niko Sünderhauf, Norman Di Palo, Nur Muhammad Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R. Sanketi, Patrick Tree Miller, Patrick Yin, Paul Wohlhart, Peng Xu 0010, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Rafael Rafailov, Ria Doshi, Roberto Martin Martin, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante-Gomez, Sean Kirmani, Sergey Levine, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham D. Sonawani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair 0003, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vincent Vanhoucke, Wolfram Burgard, Xiaolong Wang 0004, Xinghao Zhu, Xinyang Geng, Liangwei Xu, Yecheng Jason Ma 0001, Yejin Kim 0003, Yevgen Chebotar, Yilin Wu 0003, Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang 0001, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zichen Jeff Cui, Zichen Zhang 0016, Zipeng Lin |
ICRA | 214 |
| 2023 | No, to the Right: Online Language Corrections for Robotic Manipulation via Shared AutonomyabstractSystems for language-guided human-robot interaction must satisfy two key desiderata for broad adoption: adaptivity and learning efficiency. Unfortunately, existing instruction-following agents cannot adapt, lacking the ability to incorporate online natural language supervision, and even if they could, require hundreds of demonstrations to learn even simple policies. In this work, we address these problems by presenting Language-Informed Latent Actions with Corrections (LILAC), a framework for incorporating and adapting to natural language corrections "to the right", or "no, towards the book" - online, during execution. We explore rich manipulation domains within a shared autonomy paradigm. Instead of discrete turn-taking between a human and robot, LILAC splits agency between the human and robot: language is an input to a learned model that produces a meaningful, low-dimensional control space that the human can use to guide the robot. Each real-time correction refines the human's control space, enabling precise, extended behaviors - with the added benefit of requiring only a handful of demonstrations to learn. We evaluate our approach via a user study where users work with a Franka Emika Panda manipulator to complete complex manipulation tasks. Compared to existing learned baselines covering both open-loop instruction following and single-turn shared autonomy, we show that our corrections-aware approach obtains higher task completion rates, and is subjectively preferred by users because of its reliability, precision, and ease of use. Yuchen Cui, Siddharth Karamcheti, Raj Palleti, Nidhya Shivakumar, Percy Liang, Dorsa Sadigh |
HRI | 2 |
| 2023 | OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text DocumentsabstractLarge multimodal models trained on natural documents, which interleave images and text, outperform models trained on image-text pairs on various multimodal benchmarks. However, the datasets used to train these models have not been released, and the collection process has not been fully specified. We introduce the OBELICS dataset, an open web-scale filtered dataset of interleaved image-text documents comprising 141 million web pages extracted from Common Crawl, 353 million associated images, and 115 billion text tokens. We describe the dataset creation process, present comprehensive filtering rules, and provide an analysis of the dataset's content. To show the viability of OBELICS, we train on the dataset vision and language models of 9 and 80 billion parameters, IDEFICS-9B and IDEFICS, and obtain competitive performance on different multimodal benchmarks. We release our dataset, models and code. Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, Victor Sanh |
NeurIPS | 8 |
| 2021 | Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question AnsweringabstractSiddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, Christopher Manning. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021. Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei 0001, Christopher D. Manning |
ACL/IJCNLP (1) | 1 |
| 2021 | Targeted Data Acquisition for Evolving Negotiation AgentsabstractSuccessful negotiators must learn how to balance optimizing for self-interest and cooperation. Yet current artificial negotiation agents often heavily depend on the quality of the static datasets they were trained on, limiting their capacity to fashion an adaptive response balancing self-interest and cooperation. For this reason, we find that these agents can achieve either high utility or cooperation, but not both. To address this, we introduce a targeted data acquisition framework where we guide the exploration of a reinforcement learning agent using annotations from an expert oracle. The guided exploration incentivizes the learning agent to go beyond its static dataset and develop new negotiation strategies. We show that this enables our agents to obtain higher-reward and more Pareto-optimal solutions when negotiating with both simulated and human partners compared to standard supervised learning and reinforcement learning methods. This trend additionally holds when comparing agents using our targeted data acquisition framework to variants of agents trained with a mix of supervised learning and reinforcement learning, or to agents using tailored reward functions that explicitly optimize for utility and Pareto-optimality. Minae Kwon, Siddharth Karamcheti, Mariano-Florentino Cuellar, Dorsa Sadigh |
ICML | 2 |
| 2021 | ELLA: Exploration through Learned Language AbstractionabstractBuilding agents capable of understanding language instructions is critical to effective and robust human-AI collaboration. Recent work focuses on training these agents via reinforcement learning in environments with synthetic language; however, instructions often define long-horizon, sparse-reward tasks, and learning policies requires many episodes of experience. We introduce ELLA: Exploration through Learned Language Abstraction, a reward shaping approach geared towards boosting sample efficiency in sparse reward environments by correlating high-level instructions with simpler low-level constituents. ELLA has two key elements: 1) A termination classifier that identifies when agents complete low-level instructions, and 2) A relevance classifier that correlates low-level instructions with success on high-level tasks. We learn the termination classifier offline from pairs of instructions and terminal states. Notably, in departure from prior work in language and abstraction, we learn the relevance classifier online, without relying on an explicit decomposition of high-level instructions to low-level instructions. On a suite of complex BabyAI environments with varying instruction complexities and reward sparsity, ELLA shows gains in sample efficiency relative to language-based shaping and traditional RL methods. Suvir Mirchandani, Siddharth Karamcheti, Dorsa Sadigh |
NeurIPS | 2 |
| 2020 | Generating Interactive Worlds with TextabstractProcedurally generating cohesive and interesting game environments is challenging and time-consuming. In order for the relationships between the game elements to be natural, common-sense has to be encoded into arrangement of the elements. In this work, we investigate a machine learning approach for world creation using content from the multi-player text adventure game environment LIGHT (Urbanek et al. 2019). We introduce neural network based models to compositionally arrange locations, characters, and objects into a coherent whole. In addition to creating worlds based on existing elements, our models can generate new game content. Humans can also leverage our models to interactively aid in worldbuilding. We show that the game environments created with our approach are cohesive, diverse, and preferred by human evaluators compared to other machine learning based world construction algorithms. Angela Fan, Jack Urbanek, Pratik Ringshia, Emily Dinan, Emma Qian, Siddharth Karamcheti, Shrimai Prabhumoye, Douwe Kiela, Tim Rocktäschel, Arthur Szlam, Jason Weston |
AAAI | 6 |
| 2019 | Finding Generalizable Evidence by Learning to Convince Q&A ModelsabstractEthan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, Kyunghyun Cho. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019. Ethan Perez, Siddharth Karamcheti, Rob Fergus, Jason Weston, Douwe Kiela, Kyunghyun Cho |
EMNLP/IJCNLP (1) | 2 |
| 2019 | Learning to Speak and Act in a Fantasy Text Adventure GameabstractJack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, Jason Weston. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019. Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau 0001, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, Jason Weston |
EMNLP/IJCNLP (1) | 3 |