Heni Ben Amor

VLDB 2026 Research / reviewers in the wild / expert

dblp:18/3990 · DBLP profile ↗

← Back

61ranked-venue papers

6as first author

20since 2021 · last 2026

0000-0002-9896-8217ORCID · corroborated

Domains — the database's venue-derived domains; a paper can count in several

Artificial intelligence and machine learning · 50 · 5 first-author · 17 since 2021Systems, architecture and hardware · 36 · 4 first-author · 15 since 2021Human-computer interaction and ubiquitous computing · 10 · 3 since 2021Graphics, computer vision, multimedia, augmented reality and games · 7 · 1 first-authorApplied, interdisciplinary, general and emerging computing · 4 · 1 since 2021Software engineering, systems software and programming languages · 3Databases, data management, data science and information retrieval · 1Theory of computation · 1

Year	Publication	Venue	Position
2026	A Non-parametric Approach to Exploring and Quantifying the Information Flow in Human-Robot Collaboration abstract Human–Robot Interaction (HRI) has emerged as a pivotal domain in robotics, centering on the interplay and collaboration between humans and robots to achieve complex tasks. Effective communication is a cornerstone of successful HRI, facilitating the exchange of critical information essential for joint decision-making and task execution. This article explores the intricate dynamics of collaborative communication in physical HRI (pHRI), specifically focusing on non-verbal cues. Within HRI, we assert that collaboration fundamentally hinges on communication, wherein agents share information to achieve common objectives. Information theory provides a rigorous mathematical framework for quantifying the flow of information within communicating agents. It serves as a unifying framework for evaluating the dynamic interplay of various communication channels in pHRI. This study introduces a non-parametric approach based on information entropy to assess communication between agents in pHRI scenarios and detect important behaviors, such as information flow, leadership, and coupling. Through a comprehensive experimental setup involving collaborative catching tasks, we demonstrate the versatility and applicability of the proposed methodology. Gustavo Jose Giardini Lahr, Doganay Sirintuna, Francesco Tassi, Heni Ben Amor, Arash Ajoudani	ACM Trans. Hum. Robot Interact.	4
2025	SAS-Prompt: Large Language Models as Numerical Optimizers for Robot Self-Improvement abstract We demonstrate the ability of large language models (LLMs) to perform iterative self-improvement of robot policies. An important insight of this paper is that LLMs have a built-in ability to perform (stochastic) numerical optimization and that this property can be leveraged for explainable robot policy search. Based on this insight, we introduce the SAS Prompt (Summarize, Analyze, Synthesize) – a single prompt that enables iterative learning and adaptation of robot behavior by combining the LLM's ability to retrieve, reason and optimize over previous robot traces in order to synthesize new, unseen behavior. Our approach can be regarded as an early example of a new family of explainable policy search methods that are entirely implemented within an LLM. We evaluate our approach both in simulation and on a real-robot table tennis task. Project website: sites.google.com/asu.edu/sas-llm/ Heni Ben Amor, Laura Graesser, Atil Iscen, David B. D'Ambrosio, Saminda Abeyruwan, Alex Bewley, Kamalesh Kalirathinam, Swaroop Mishra, Pannag R. Sanketi	ICRA	1
2025	Achieving Human Level Competitive Robot Table Tennis abstract Achieving human-level performance on real world tasks is a north star for the robotics community. We present the first learned robot agent that reaches amateur humanlevel performance in competitive table tennis. Table tennis is a physically demanding sport that takes humans years to master. We contribute (1) a hierarchical and modular policy architecture consisting of (i) low level controllers with their skill descriptors that model their capabilities and (ii) a high level controller that chooses the low level skills, (2) techniques for enabling zero-shot sim-to-real and curriculum building, including an iterative approach (train in sim, deploy in real), and (3) real time adaptation to unseen opponents. Policy performance was assessed through 29 robot vs. human matches of which the robot won 45 % (13/29). All humans were unseen players and their skill level varied from beginner to tournament level. Whilst the robot lost all matches vs. the most advanced players it won 100 % matches vs. beginners and 55 % matches vs. intermediate players, demonstrating solidly amateur humanlevel performance. Videos of the matches can be viewed here1.See sites https://google.com/view/competitive-robot-table-tennis. David B. D'Ambrosio, Saminda Abeyruwan, Laura Graesser, Atil Iscen, Heni Ben Amor, Alex Bewley, Barney J. Reed, Krista Reymann, Leila Takayama, Yuval Tassa, Krzysztof Choromanski, Erwin Coumans, Deepali Jain, Navdeep Jaitly, Natasha Jaques, Satoshi Kataoka, Yuheng Kuang, Nevena Lazic, Reza Mahjourian, Sherry Moore, Kenneth Oslund, Anish Shankar, Vikas Sindhwani, Vincent Vanhoucke, Grace Vesom, Peng Xu 0010, Pannag R. Sanketi	ICRA	5
2025	Uncertainty-aware Motion Planning based on Stochastic Forward/Inverse Kinematics Models for Tensegrity Manipulators abstract Robots whose shape and stiffness are determined by internal forces generally have complex shape-stiffness relationships that depend on their structure. As a result, there are difficulties such as a decrease in shape reproducibility when the robot is not stiff, and a decrease in the range of motion when the robot is stiff. In this study, we propose a motion planning method that balances shape and stiffness by learning forward and inverse kinematics using a stochastic neural network (NN) and using the uncertainty that can be evaluated by the NN. Through experiments using a tensegrity manipulator with 40 actuators and 20 degrees of freedom in bending posture, we verify the validity of the proposed method. Yuhei Yoshimitsu, Takayuki Osa, Heni Ben Amor, Shuhei Ikemoto	IROS	3
2025	Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs abstract Reinforcement Learning (RL) traditionally relies on scalar reward signals, limiting its ability to leverage the rich semantic knowledge often available in real-world tasks. In contrast, humans learn efficiently by combining numerical feedback with language, prior knowledge, and common sense. We introduce Prompted Policy Search (ProPS), a novel RL method that unifies numerical and linguistic reasoning within a single framework. Unlike prior work that augments existing RL components with language, ProPS places a large language model (LLM) at the center of the policy optimization loop—directly proposing policy updates based on both reward feedback and natural language input. We show that LLMs can perform numerical optimization in-context, and that incorporating semantic signals, such as goals, constraints, and strategy hints can lead to more informed exploration and sample-efficient learning. ProPS is evaluated across 15 Gymnasium tasks, spanning classic control, Atari games, and MuJoCo environments, and compared to seven widely-adopted RL algorithms (e.g., PPO, SAC, TRPO). It outperforms all baselines on 8 out of 15 tasks and demonstrates substantial gains when provided with domain knowledge. These results highlight the potential of unifying semantics and numerics for transparent, generalizable, and human-aligned reinforcement learning. Sachin Grover, Mohamed El Mistiri, Kamalesh Kalirathinam, Pratyush Kerhalkar, Swaroop Mishra, Sanket Gaurav, Oya Aran, Heni Ben Amor	NeurIPS	10
2024	Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration abstract Large, high-capacity models trained on diverse datasets have shown remarkable successes on efficiently tackling downstream applications. In domains from NLP to Computer Vision, this has led to a consolidation of pretrained models, with general pretrained backbones serving as a starting point for many applications. Can such a consolidation happen in robotics? Conventionally, robotic learning methods train a separate model for every application, every robot, and even every environment. Can we instead train "generalist" X-robot policy that can be adapted efficiently to new robots, tasks, and environments? In this paper, we provide datasets in standardized data formats and models to make it possible to explore this possibility in the context of robotic manipulation, alongside experimental results that provide an example of effective X-robot policies. We assemble a dataset from 22 different robots collected through a collaboration between 21 institutions, demonstrating 527 skills (160266 tasks). We show that a high-capacity model trained on this data, which we call RT-X, exhibits positive transfer and improves the capabilities of multiple robots by leveraging experience from other platforms. The project website is robotics-transformer-x.github.io. Abigail O'Neill, Abhiram Maddukuri, Abhishek Gupta 0004, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, Albert Tung, Alex Bewley, Alex Irpan, Alexander Khazatsky, Anant Rai, Anchit Gupta, Andrew E. Wang, Anikait Singh, Animesh Garg, Aniruddha Kembhavi, Annie Xie, Anthony Brohan, Antonin Raffin, Archit Sharma, Arefeh Yavary, Arhan Jain, Ashwin Balakrishna, Ayzaan Wahid, Ben Burgess-Limerick, Bernhard Schölkopf, Blake Wulfe, Brian Ichter, Cewu Lu, Charles Xu 0003, Charlotte Le, Chelsea Finn, Chen Wang 0053, Chenfeng Xu, Cheng Chi 0001, Chenguang Huang, Christine Chan, Christopher Agia, Chuer Pan, Chuyuan Fu, Coline Devin, Danfei Xu, Daniel Morton, Danny Drieß, Daphne Chen, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dinesh Jayaraman, Dmitry Kalashnikov, Dorsa Sadigh, Edward Johns, Ethan Paul Foster, Fangchen Liu, Federico Ceola, Fei Xia 0002, Feiyu Zhao, Freek Stulp, Gaoyue Zhou, Gaurav S. Sukhatme, Gautam Salhotra, Gilbert Feng, Giulio Schiavi, Glen Berseth, Gregory Kahn, Guanzhi Wang, Hao Su 0001, Haoshu Fang, Henghui Bao, Heni Ben Amor, Henrik I. Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Huy Ha, Igor Mordatch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jad Abou-Chakra, Jaehyung Kim 0001, Jaimyn Drake, Jan Peters 0001, Jan Schneider 0007, Jasmine Hsu, Jeannette Bohg, Jeffrey T. Bingham, Jensen Gao, Jiaheng Hu, Jiajun Wu 0001, Jiankai Sun, Jianlan Luo, Jiayuan Gu, Jie Tan 0001, Jihoon Oh, Jimmy Wu, Jingpei Lu, Jitendra Malik, João Silvério, Joey Hejna, Jonathan Booher, Jonathan Tompson, Jonathan Yang, Jordi Salvador, Joseph J. Lim, Junhyek Han, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Kenneth Y. Goldberg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Black, Kevin Zhang 0002, Kiana Ehsani, Kiran Lekkala, Kirsty Ellis, Krishan Rana, Krishnan Srinivasan, Kuan Fang, Kunal Pratap Singh, Kuo-Hao Zeng, Kyle Hatch, Kyle Hsu, Laurent Itti, Yunliang Chen 0001, Lerrel Pinto, Li Fei-Fei 0001, Liam Tan, Linxi Fan, Lionel Ott, Lisa Lee, Luca Weihs, Magnum Chen, Marion Lepert, Marius Memmel, Masayoshi Tomizuka, Masha Itkina, Mateo Guaman Castro, Max Spero, Maximilian Du, Michael Ahn, Michael C. Yip, Mingtong Zhang 0003, Mingyu Ding, Minho Heo, Mohan Kumar Srirama, Mohit Sharma 0001, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen 0001, Nicolas Heess, Nikhil J. Joshi, Niko Sünderhauf, Norman Di Palo, Nur Muhammad Shafiullah, Oier Mees, Oliver Kroemer, Osbert Bastani, Pannag R. Sanketi, Patrick Tree Miller, Patrick Yin, Paul Wohlhart, Peng Xu 0010, Peter David Fagan, Peter Mitrano, Pierre Sermanet, Pieter Abbeel, Priya Sundaresan, Qiuyu Chen, Rafael Rafailov, Ria Doshi, Roberto Martin Martin, Rohan Baijal, Rosario Scalise, Rose Hendrix, Roy Lin, Runjia Qian, Russell Mendonca, Rutav Shah, Ryan Hoque, Ryan Julian, Samuel Bustamante-Gomez, Sean Kirmani, Sergey Levine, Sherry Moore, Shikhar Bahl, Shivin Dass, Shubham D. Sonawani, Shuran Song, Sichun Xu, Siddhant Haldar, Siddharth Karamcheti, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Subramanian Ramamoorthy, Sudeep Dasari, Suneel Belkhale, Sungjae Park, Suraj Nair 0003, Suvir Mirchandani, Takayuki Osa, Tanmay Gupta, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Thomas Kollar, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Z. Zhao, Travis Armstrong, Trevor Darrell, Trinity Chung, Vidhi Jain, Vincent Vanhoucke, Wolfram Burgard, Xiaolong Wang 0004, Xinghao Zhu, Xinyang Geng, Liangwei Xu, Yecheng Jason Ma 0001, Yejin Kim 0003, Yevgen Chebotar, Yilin Wu 0003, Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yueh-Hua Wu, Yujin Tang, Yuke Zhu, Yunchu Zhang, Yunfan Jiang 0001, Yunshuang Li, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zehan Ma, Zichen Jeff Cui, Zichen Zhang 0016, Zipeng Lin	ICRA	78
2024	iRoCo: Intuitive Robot Control From Anywhere Using a Smartwatch abstract This paper introduces iRoCo (intuitive Robot Control) – a framework for ubiquitous human-robot collaboration using a single smartwatch and smartphone. By integrating probabilistic differentiable filters, iRoCo optimizes a combination of precise robot control and unrestricted user movement from ubiquitous devices. We demonstrate and evaluate the effectiveness of iRoCo in practical teleoperation and drone piloting applications. Comparative analysis shows no significant difference between task performance with iRoCo and gold-standard control systems in teleoperation tasks. Additionally, iRoCo users complete drone piloting tasks 32% faster than with a traditional remote control and report less frustration in a subjective load index questionnaire. Our findings strongly suggest that iRoCo is a promising new approach for intuitive robot control through smartwatches and smart-phones from anywhere, at any time. The code is available at www.github.com/wearable-motion-capture Fabian Clemens Weigend, Shubham D. Sonawani, Venu Vasudevan, Heni Ben Amor	ICRA	6
2024	Diff-Control: A Stateful Diffusion-based Policy for Imitation Learning abstract While imitation learning provides a simple and effective framework for policy learning, acquiring consistent action during robot execution remains a challenging task. Existing approaches primarily focus on either modifying the action representation at data curation stage or altering the model itself, both of which do not fully address the scalability of consistent action generation. To overcome this limitation, we introduce the Diff-Control policy, which utilizes a diffusion-based model to learn action representation from a state-space modeling viewpoint. We demonstrate that diffusion-based policies can acquire statefulness through a Bayesian formulation facilitated by ControlNet, leading to improved robustness and success rates. Our experimental results demonstrate the significance of incorporating action statefulness in policy learning, where Diff-Control shows improved performance across various tasks. Specifically, Diff-Control achieves an average success rate of 72% and 84% on stateful and dynamic tasks, respectively. Notably, Diff-Control also shows consistent performance in the presence of perturbations, outperforming other state-of-the-art methods that falter under similar conditions. Project page: https://diff-control.github.io/ Fabian Clemens Weigend, Shubham D. Sonawani, Shuhei Ikemoto, Heni Ben Amor	IROS	6
2024	Repairing Neural Networks for Safety in Robotic Systems using Predictive Models abstract This paper introduces a new method for safety-aware robot learning, focusing on repairing policies using predictive models. Our method combines behavioral cloning with neural network repair in a two-step supervised learning framework. It first learns a policy from expert demonstrations and then applies repair subject to predictive models to enforce safety constraints. The predictive models can encompass various aspects relevant to robot learning applications, such as proprioceptive states and collision likelihood. Our experimental results demonstrate that the learned policy successfully adheres to a predefined set of safety constraints on two applications: mobile robot navigation, and real-world lower-leg prostheses. Additionally, we have shown that our method effectively reduces repeated interaction with the robot, leading to substantial time savings during the learning process. Keyvan Majd, Geoffrey Clark, Georgios Fainekos, Heni Ben Amor	IROS	4
2024	SiSCo: Signal Synthesis for Effective Human-Robot Communication Via Large Language Models abstract Effective human-robot collaboration hinges on robust communication channels, with visual signaling playing a pivotal role due to its intuitive appeal. Yet, the creation of visually intuitive cues often demands extensive resources and specialized knowledge. The emergence of Large Language Models (LLMs) offers promising avenues for enhancing human-robot interactions and revolutionizing the way we generate context-aware visual cues. To this end, we introduce SiSCo–a novel framework that combines the computational power of LLMs with mixed-reality technologies to streamline the creation of visual cues for human-robot collaboration. Our results show that SiSCo improves the efficiency of communication in human-robot teaming tasks, reducing task completion time by approximately 73% and increasing task success rates by 18% compared to baseline natural language signals. Additionally, SiSCo reduces cognitive load for participants by 46%, as measured by the NASA-TLX subscale, and receives above-average user ratings for on-the-fly signals generated for unseen objects. To encourage further development and broader community engagement, we provide full access to SiSCo’s implementation and related materials on our GitHub repository.1 Shubham D. Sonawani, Fabian Clemens Weigend, Heni Ben Amor	IROS	3
2024	Active Learning for Forward/Inverse Kinematics of Redundantly-driven Flexible Tensegrity Manipulator abstract In flexible redundantly-driven multi-DOF systems, like living beings, the representation of redundant kinematics including the diversity of solutions, is crucial for leveraging its distinctive characteristics. This paper proposes an active learning framework for forward and inverse modeling of complex kinematics that improves expressions of control space, task space, and null space. It consists of a Variational Auto Encoder (VAE)-type network that internally holds expressions of control space, task space, and null space, and an algorithm for selecting new data using the cross-entropy method. The validity of the proposed system was verified using a tensegrity manipulator driven by 40 pneumatic cylinders. As a result, it was confirmed that active learning contributed to achieving the entire range of motion covered and a well-organized representation of the null space. Yuhei Yoshimitsu, Takayuki Osa, Heni Ben Amor, Shuhei Ikemoto	IROS	3
2023	Learning and Blending Robot Hugging Behaviors in Time and Space abstract We introduce an imitation learning-based physical human-robot interaction algorithm capable of predicting appropriate robot responses in complex interactions involving a superposition of multiple interactions. Our proposed algorithm, Blending Bayesian Interaction Primitives (B-BIP) allows us to achieve responsive interactions in complex hugging scenarios, capable of reciprocating and adapting to a hug's motion and timing. We show that this algorithm is a generalization of prior work, for which the original formulation reduces to the particular case of a single interaction, and evaluate our method through both an extensive user study and empirical experiments. Our algorithm yields significantly better quantitative prediction error and more-favorable participant responses with respect to accuracy, responsiveness, and timing, when compared to existing state-of-the-art methods. Michael Drolet, Joseph Campbell, Heni Ben Amor	ICRA	3
2023	Enhancing State Estimation in Robots: A Data-Driven Approach with Differentiable Ensemble Kalman Filters abstract This paper introduces a novel state estimation framework for robots using differentiable ensemble Kalman filters (DEnKF). DEnKF is a reformulation of the traditional ensemble Kalman filter that employs stochastic neural networks to model the process noise implicitly. Our work is an extension of previous research on differentiable filters, which has provided a strong foundation for our modular and end-to-end differentiable framework. This framework enables each component of the system to function independently, leading to improved flexibility and versatility in implementation. Through a series of experiments, we demonstrate the flexibility of this model across a diverse set of real-world tracking tasks, including visual odometry and robot manipulation. Moreover, we show that our model effectively handles noisy observations, is robust in the absence of observations, and outperforms state-of-the-art differentiable filters in terms of error metrics. Specifically, we observe a significant improvement of at least 59% in translational error when using DEnKF with noisy observations. Our results underscore the potential of DEnKF in advancing state estimation for robotics. Code for DEnKF is available at https://github.com/ir-lab/DEnKF Geoffrey Clark, Joseph Campbell, Heni Ben Amor	IROS	5
2023	Learning Soft Robot Dynamics Using Differentiable Kalman Filters and Spatio-Temporal Embeddings abstract This paper introduces a novel approach for modeling the dynamics of soft robots, utilizing a differentiable filter architecture. The proposed approach enables end-to-end training to learn system dynamics, noise characteristics, and temporal behavior of the robot. A novel spatio-temporal embedding process is discussed to handle observations with varying sensor placements and sampling frequencies. The efficacy of this approach is demonstrated on a tensegrity robot arm by learning end-effector dynamics from demonstrations with complex bending motions. The model is proven to be robust against missing modalities, diverse sensor placement, and varying sampling rates. Additionally, the proposed framework is shown to identify physical interactions with humans during motion. The utilization of a differentiable filter presents a novel solution to the difficulties of modeling soft robot dynamics. Our approach shows substantial improvement in accuracy compared to state-of-the-art filtering methods, with at least a 24% reduction in mean absolute error (MAE) observed. Furthermore, the predicted end-effector positions show an average MAE of 25.77mm from the ground truth, highlighting the advantage of our approach. The code is available at https://github.com/ir-lab/soft_robot_DEnKF. Shuhei Ikemoto, Yuhei Yoshimitsu, Heni Ben Amor	IROS	4
2023	Projecting Robot Intentions Through Visual Cues: Static vs. Dynamic Signaling abstract Augmented and mixed-reality techniques harbor a great potential for improving human-robot collaboration. Visual signals and cues may be projected to a human partner in order to explicitly communicate robot intentions and goals. However, it is unclear what type of signals support such a process and whether signals can be combined without adding additional cognitive stress to the partner. This paper focuses on identifying the effective types of visual signals and quantify their impact through empirical evaluations. In particular, the study compares static and dynamic visual signals within a collaborative object sorting task and assesses their ability to shape human behavior. Furthermore, an information-theoretic analysis is performed to numerically quantify the degree of information transfer between visual signals and human behavior. The results of a human subject experiment show that there are significant advantages to combining multiple visual signals within a single task, i.e., increased task efficiency and reduced cognitive load. Shubham D. Sonawani, Heni Ben Amor	IROS	3
2023	Anytime, Anywhere: Human Arm Pose from Smartwatch Data for Ubiquitous Robot Control and Teleoperation abstract This work devises an optimized machine learning approach for human arm pose estimation from a single smart-watch. Our approach results in a distribution of possible wrist and elbow positions, which allows for a measure of uncertainty and the detection of multiple possible arm posture solutions, i.e., multimodal pose distributions. Combining estimated arm postures with speech recognition, we turn the smartwatch into a ubiquitous, low-cost and versatile robot control interface. We demonstrate in two use-cases that this intuitive control interface enables users to swiftly intervene in robot behavior, to temporarily adjust their goal, or to train completely new control policies by imitation. Extensive experiments show that the approach results in a 40% reduction in prediction error over the current state-of-the-art and achieves a mean error of 2.56 cm for wrist and elbow positions. Fabian Clemens Weigend, Shubham D. Sonawani, Michael Drolet, Heni Ben Amor	IROS	4
2023	Learning Ergonomic Control in Human-Robot Symbiotic Walking abstract This article presents an imitation learning strategy for extracting ergonomically safe control policies in physical human–robot interaction scenarios. The presented approach seeks to proactively reduce the risk of injuries and musculoskeletal disorders by anticipating the ergonomic effects of a robot's actions on a human partner, e.g., how the ankle angle of a prosthesis affects future knee torques of the user. To this end, we extend ensemble Bayesian interaction primitives to enable the prediction of latent biomechanical variables. This methodology yields a reactive control strategy, which we evaluate in an assisted walking task with a robotic lower limb prosthesis. Building upon the learned interaction primitives, we also present a model-predictive control (MPC) strategy that actively steers the human–robot interaction toward ergonomic and safe movement regimes. We compare the introduced control strategies and highlight the framework's ability to generate ergonomic, biomechanically safe assistive prosthetic control. A rich analysis of constrained MPC shows a 20× reduction in the effects of large perturbations on prosthetic control system. We empirically demonstrate a 16% reduction in vertical knee reaction forces in real-world jumping experiments utilizing our control methodology and examine other optimal control strategies in simulated walking experiments. Geoffrey Clark, Heni Ben Amor	IEEE Trans. Robotics	2
2022	4th Annual Workshop on Test Methods and Metrics for Effective HRI abstract The drive for increasing adoption of HRI technolo-gies is evident through research and development of manufac-turing, social, medical, and service robot solutions. However, novel methods and metrics are required to overcome the barrier between fundamental HRI research and its adoption in real-world environments. Hence, the fourth installment of the annual workshop, 'Test Methods and Metrics for Effective HRI,’ seeks to identify novel and emerging test methods and metrics for the holistic assessment and assurance of HRI performance. Specifically, the focus is on identifying innovative methods for the evaluation of HRI performance and to advance the growth of the HRI community based on the principles of collaboration, data sharing, and repeatability. The goal of this workshop is to break the boundaries between the development and adoption of HRI technologies through the promotion of robust experimental design, test methods, and metrics for assessing interaction and interface designs. This workshop will have participants from var-ious sectors in the HRI research community including academia, industry, and government in order to accomplish its aims. Jeremy A. Marvel, Shelly Bagchi, Megan Zimmerman, Vinh Nguyen 0001, Murat Aksu, Brian Antonishek, Jennifer C. Case, Heni Ben Amor, Terrence Fong, Ross Mead, Adam Norton, Yue Wang 0011	HRI	8
2022	A System for Imitation Learning of Contact-Rich Bimanual Manipulation Policies abstract In this paper, we discuss a framework for teaching bimanual manipulation tasks by imitation. To this end, we present a system and algorithms for learning compliant and contact-rich robot behavior from human demonstrations. The presented system combines insights from admittance control and machine learning to extract control policies that can (a) recover from and adapt to a variety of disturbances in time and space, while also (b) effectively leveraging physical contact with the environment. We demonstrate the effectiveness of our approach using a real-world insertion task involving multiple simultaneous contacts between a manipulated object and insertion pegs. We also investigate efficient means of collecting training data for such bimanual settings. To this end, we conduct a human-subject study and analyze the effort and mental demand as reported by the users. Our experiments show that, while harder to provide, the additional force/torque information available in teleoperated demonstrations is crucial for phase estimation and task success. Ultimately, force/torque data substantially improves manipulation robustness, resulting in a 90% success rate in a multipoint insertion task. Code and videos can be found at https://bimanualmanipulation.com/ Simon Stepputtis, Maryam Bandari, Stefan Schaal, Heni Ben Amor	IROS	4
2022	Introduction to the Special Issue on Test Methods for Human-Robot Teaming Performance Evaluations abstract This special issue of the Transactions on Human-Robot Interaction highlights, documents, and explores the metrics, test methods, and artifacts used in human-robot interaction (HRI) research. This collection of articles brings to attention the commonalities between the application of measurement science for the assessment and assurance of human-centric robotics in a variety of application domains, including industry, education, and defense. This special issue draws specific attention to the use and impact of metrology toward the advancement of HRI technologies and algorithms, and it promotes the application of measurement science toward the benchmarking and replication of HRI research. Special attention is given to the use cases, data sets, test methodologies, measurement techniques, metrics, and statistical analyses used to evaluate system performance. Jeremy A. Marvel, Shelly Bagchi, Megan Zimmerman, Murat Aksu, Brian Antonishek, Yue Wang 0011, Ross Mead, Terrence Fong, Heni Ben Amor	ACM Trans. Hum. Robot Interact.	9
2020	DeepCrashTest: Turning Dashcam Videos into Virtual Crash Tests for Automated Driving Systems abstract The goal of this paper is to generate simulations with real-world collision scenarios for training and testing autonomous vehicles. We use numerous dashcam crash videos uploaded on the internet to extract valuable collision data and recreate the crash scenarios in a simulator. We tackle the problem of extracting 3D vehicle trajectories from videos recorded by an unknown and uncalibrated monocular camera source using a modular approach. A working architecture and demonstration videos along with the open-source implementation are provided with the paper. Sai Krishna Bashetty, Heni Ben Amor, Georgios Fainekos	ICRA	2
2020	Predictive Modeling of Periodic Behavior for Human-Robot Symbiotic Walking abstract We propose in this paper Periodic Interaction Primitives - a probabilistic framework that can be used to learn compact models of periodic behavior. Our approach extends existing formulations of Interaction Primitives to periodic movement regimes, i.e., walking. We show that this model is particularly well-suited for learning data-driven, customized models of human walking, which can then be used for generating predictions over future states or for inferring latent, biomechanical variables. We also demonstrate how the same framework can be used to learn controllers for a robotic prosthesis using an imitation learning approach. Results in experiments with human participants indicate that Periodic Interaction Primitives efficiently generate predictions and ankle angle control signals for a robotic prosthetic ankle, with MAE of 2.21° in 0.0008s per inference. Performance degrades gracefully in the presence of noise or sensor fall outs. Compared to alternatives, this algorithm functions 20 times faster and performed 4.5 times more accurately on test subjects. Geoffrey Clark, Joseph Campbell, Seyed Mostafa Rezayat Sorkhabadi, Heni Ben Amor	ICRA	5
2020	Language-Conditioned Imitation Learning for Robot Manipulation Tasks abstract Imitation learning is a popular approach for teaching motor skills to robots. However, most approaches focus on extracting policy parameters from execution traces alone (i.e., motion trajectories and perceptual data). No adequate communication channel exists between the human expert and the robot to describe critical aspects of the task, such as the properties of the target object or the intended shape of the motion. Motivated by insights into the human teaching process, we introduce a method for incorporating unstructured natural language into imitation learning. At training time, the expert can provide demonstrations along with verbal descriptions in order to describe the underlying intent (e.g., "go to the large green bowl"). The training process then interrelates these two modalities to encode the correlations between language, perception, and motion. The resulting language-conditioned visuomotor policies can be conditioned at runtime on new human commands and instructions, which allows for more fine-grained control over the trained policies while also reducing situational ambiguity. We demonstrate in a set of simulation experiments how our approach can learn language-conditioned manipulation policies for a seven-degree-of-freedom robot arm and compare the results to a variety of alternative methods. Simon Stepputtis, Joseph Campbell, Mariano J. Phielipp, Stefan Lee, Chitta Baral, Heni Ben Amor	NeurIPS	6
2020	Comparison of novelty detection methods for multispectral images in rover-based planetary exploration missions abstract Abstract Science teams for rover-based planetary exploration missions like the Mars Science Laboratory Curiosity rover have limited time for analyzing new data before making decisions about follow-up observations. There is a need for systems that can rapidly and intelligently extract information from planetary instrument datasets and focus attention on the most promising or novel observations. Several novelty detection methods have been explored in prior work for three-channel color images and non-image datasets, but few have considered multispectral or hyperspectral image datasets for the purpose of scientific discovery. We compared the performance of four novelty detection methods—Reed Xiaoli (RX) detectors, principal component analysis (PCA), autoencoders, and generative adversarial networks (GANs)—and the ability of each method to provide explanatory visualizations to help scientists understand and trust predictions made by the system. We show that pixel-wise RX and autoencoders trained with structural similarity (SSIM) loss can detect morphological novelties that are not detected by PCA, GANs, and mean squared error autoencoders, but that the latter methods are better suited for detecting spectral novelties—i.e., the best method for a given setting depends on the type of novelties that are sought. Additionally, we find that autoencoders provide the most useful explanatory visualizations for enabling users to understand and trust model detections, and that existing GAN approaches to novelty detection may be limited in this respect. Hannah Kerner, Kiri Wagstaff, Brian D. Bue, Danika F. Wellington, Samantha Jacob, Paul Horton, James F. Bell, Chiman Kwan, Heni Ben Amor	Data Min. Knowl. Discov.	9
2019	Novelty Detection for Multispectral Images with Application to Planetary Exploration abstract In this work, we present a system based on convolutional autoencoders for detecting novel features in multispectral images. We introduce SAMMIE: Selections based on Autoencoder Modeling of Multispectral Image Expectations. Previous work using autoencoders employed the scalar reconstruction error to classify new images as novel or typical. We show that a spatial-spectral error map can enable both accurate classification of novelty in multispectral images as well as human-comprehensible explanations of the detection. We apply our methodology to the detection of novel geologic features in multispectral images of the Martian surface collected by the Mastcam imaging system on the Mars Science Laboratory Curiosity rover. Hannah Kerner, Danika F. Wellington, Kiri Wagstaff, James F. Bell, Chiman Kwan, Heni Ben Amor	AAAI	6
2019	Specifying and Evaluating Quality Metrics for Vision-based Perception Systems abstract Robust perception algorithms are a vital ingredient for autonomous systems such as self-driving vehicles. Checking the correctness of perception algorithms such as those based on deep convolutional neural networks (CNN) is a formidable challenge problem. In this paper, we suggest the use of Timed Quality Temporal Logic (TQTL) as a formal language to express desirable spatio-temporal properties of a perception algorithm processing a video. While perception algorithms are traditionally tested by comparing their performance to ground truth labels, we show how TQTL can be a useful tool to determine quality of perception, and offers an alternative metric that can give useful information, even in the absence of ground truth labels. We demonstrate TQTL monitoring on two popular CNNs: YOLO and SqueezeDet, and give a comparative study of the results obtained for each architecture. Anand Balakrishnan 0001, Aniruddh Gopinath Puranic, Adel Dokhanchi, Jyotirmoy V. Deshmukh, Heni Ben Amor, Georgios Fainekos	DATE	6
2019	Test Methods and Metrics for Effective HRI in Collaborative Human-Robot Teams abstract Verified and validated test methods, being necessary to measure the performance of complex systems, are important tools for driving innovation, benchmarking and improving performance, and establishing trust in collaborative human-robot teams. This full-day workshop aims to explore the metrology necessary for repeatably and independently assessing the collaborative performance of robotic systems in real-world human-robot interaction (HRI) scenarios. This workshop aims to bridge the gaps between the theory and applications of HRI in industry, accelerating the adoption of cutting edge technologies as the industry state-of-practice. The interest in collaborative HRI is evident in the current market as well as standards efforts toward manufacturing, social, medical, and service robot solutions. Though these domains have been considered separate for many years, recent technological and scientific advancements show that, while their applications may differ, the underlying principles of HRI performance impact each identically. As such, this workshop seeks to identify test methods and metrics for the holistic assessment and assurance of collaborative HRI performance. The focus is on identifying the key performance indicators of these seemingly disparate sectors, and additionally to establish a community based on the principles of transparency, repeatability, & establishing trust in the assessment of collaborative HRI. The goal is to aid in the advancement of HRI technologies through the development of experimental scenarios, protocols, test methods, & metrics for the verification and validation of interaction solutions and interface designs. Jeremy A. Marvel, Shelly Bagchi, Megan Zimmerman, Murat Aksu, Brian Antonishek, Yue Wang 0011, Ross Mead, Terrence Fong, Heni Ben Amor	HRI	9
2019	Deep Learning of Proprioceptive Models for Robotic Force Estimation abstract Many robotic tasks require fast and accurate force sensing capabilities to ensure adaptive behavior execution. While dedicated force-torque (FT) sensors are a common option, such devices induce extra costs, need additional power supply, and add weight to otherwise light-weight robotic systems. This paper presents a machine learning approach for estimating external forces acting on a robot based on common internal sensors only. In the training phase, a behavior-specific proprioceptive model is learned as compact representation of the expected proprioceptive feedback during task execution. First, the proprioceptive sensors relevant for the given behavior are identified using information-theoretic measures. Then, the proprioceptive model is learned using deep learning techniques. During behavior execution, the proprioceptive model is applied to actual sensor readings for estimation of external forces. Experiments performed with the UR5 robot demonstrate the ability for fast and accurate force estimation even in situations where a dedicated commercial FT sensor is not applicable. Erik Berger, Daniel Eger Passos, Steve Grehl, Heni Ben Amor, Bernhard Jung 0001	IROS	4
2019	Learning Interactive Behaviors for Musculoskeletal Robots Using Bayesian Interaction Primitives abstract Musculoskeletal robots that are based on pneumatic actuation have a variety of properties, such as compliance and back-drivability, that render them particularly appealing for human-robot collaboration. However, programming interactive and responsive behaviors for such systems is extremely challenging due to the nonlinearity and uncertainty inherent to their control. In this paper, we propose an approach for learning Bayesian Interaction Primitives for musculoskeletal robots given a limited set of example demonstrations. We show that this approach is capable of real-time state estimation and response generation for interaction with a robot for which no analytical model exists. Human-robot interaction experiments on a 'handshake' task show that the approach generalizes to new positions, interaction partners, and movement velocities. Joseph Campbell, Arne Hitzmann, Simon Stepputtis, Shuhei Ikemoto, Koh Hosoda, Heni Ben Amor	IROS	6
2019	Improved Exploration through Latent Trajectory Optimization in Deep Deterministic Policy Gradient abstract Model-free reinforcement learning algorithms such as Deep Deterministic Policy Gradient (DDPG) often require additional exploration strategies, especially if the actor is of deterministic nature. This work evaluates the use of model-based trajectory optimization methods used for exploration in Deep Deterministic Policy Gradient when trained on a latent image embedding. In addition, an extension of DDPG is derived using a value function as critic, making use of a learned deep dynamics model to compute the policy gradient. This approach leads to a symbiotic relationship between the deep reinforcement learning algorithm and the latent trajectory optimizer. The trajectory optimizer benefits from the critic learned by the RL algorithm and the latter from the enhanced exploration generated by the planner. The developed methods are evaluated on two continuous control tasks, one in simulation and one in the real world. In particular, a Baxter robot is trained to perform an insertion task, while only receiving sparse rewards and images as observations from the environment. Kevin S. Luck, Mel Vecerík, Simon Stepputtis, Heni Ben Amor, Jonathan Scholz	IROS	4
2019	Clone Swarms: Learning to Predict and Control Multi-Robot Systems by Imitation abstract In this paper, we propose SwarmNet - a neural network architecture that can learn to predict and imitate the behavior of an observed swarm of agents in a centralized manner. Tested on artificially generated swarm motion data, the network achieves high levels of prediction accuracy and imitation authenticity. We compare our model to previous approaches for modelling interaction systems and show how modifying components of other models gradually approaches the performance of ours. Finally, we also discuss an extension of SwarmNet that can deal with nondeterministic, noisy, and uncertain environments, as often found in robotics applications. Siyu Zhou 0001, Mariano J. Phielipp, Jorge A. Sefair, Sara Imari Walker, Heni Ben Amor	IROS	5
2019	Encoding and monitoring responsibility sensitive safety rules for automated vehicles in signal temporal logic abstract As Automated Vehicles (AV) get ready to hit the public roads unsupervised, many practical questions still remain open. For example, there is no commonly acceptable formal definition of what safe driving is. A formal definition of safe driving can be utilized in developing the vehicle behaviors as well as in certification and legal cases. Toward that goal, the Responsibility-Sensitive Safety (RSS) model was developed as a first step toward formalizing safe driving behavior upon which the broader AV community can expand. In this paper, we demonstrate that the RSS model can be encoded in Signal Temporal Logic (STL). Moreover, using the S-TaLiRo tools, we present a case study of monitoring RSS requirements on selected traffic scenarios from CommonRoad. We conclude that monitoring RSS rules encoded in STL is efficient even in heavy traffic scenarios. One interesting observation is that for the selected traffic data, vehicle parameters and response times, the RSS model violations are not frequent. Mohammad Hekmatnejad, Shakiba Yaghoubi, Adel Dokhanchi, Heni Ben Amor, Aviral Shrivastava, Lina J. Karam, Georgios Fainekos	MEMOCODE	4
2018	Extrinsic Dexterity Through Active Slip Control Using Deep Predictive Models abstract We present a machine learning methodology for actively controlling slip, in order to increase robot dexterity. Leveraging recent insights in deep learning, we propose a Deep Predictive Model that uses tactile sensor information to reason about slip and its future influence on the manipulated object. The obtained information is then used to precisely manipulate objects within a robot end-effector using external perturbations imposed by gravity or acceleration. We show in a set of experiments that this approach can be used to increase a robot's repertoire of motor skills. Simon Stepputtis, Yezhou Yang, Heni Ben Amor	ICRA	3
2018	Deep Predictive Models for Collision Risk Assessment in Autonomous Driving abstract In this paper, we investigate a predictive approach for collision risk assessment in autonomous and assisted driving. A deep predictive model is trained to anticipate imminent accidents from traditional video streams. In particular, the model learns to identify cues in RGB images that are predictive of hazardous upcoming situations. In contrast to previous work, our approach incorporates (a) temporal information during decision making, (b) multi-modal information about the environment, as well as the proprioceptive state and steering actions of the controlled vehicle, and (c) information about the uncertainty inherent to the task. To this end, we discuss Deep Predictive Models and present an implementation using a Bayesian Convolutional LSTM. Experiments in a simple simulation environment show that the approach can learn to predict impending accidents with reasonable accuracy, especially when multiple cameras are used as input sources. Mark Strickland, Georgios Fainekos, Heni Ben Amor	ICRA	3
2018	Evaluating Perception Systems for Autonomous Vehicles Using Quality Temporal Logic Adel Dokhanchi, Heni Ben Amor, Jyotirmoy V. Deshmukh, Georgios Fainekos	RV	2
2017	A system for learning continuous human-robot interactions from human-human demonstrations abstract We present a data-driven imitation learning system for learning human-robot interactions from human-human demonstrations. During training, the movements of two interaction partners are recorded through motion capture and an interaction model is learned. At runtime, the interaction model is used to continuously adapt the robot's motion, both spatially and temporally, to the movements of the human interaction partner. We show the effectiveness of the approach on complex, sequential tasks by presenting two applications involving collaborative human-robot assembly. Experiments with varied object hand-over positions and task execution speeds confirm the capabilities for spatio-temporal adaption of the demonstrated behavior to the current situation. David Vogt, Simon Stepputtis, Steve Grehl, Bernhard Jung 0001, Heni Ben Amor	ICRA	5
2017	Extracting bimanual synergies with reinforcement learning abstract Motor synergies are an important concept in human motor control. Through the co-activation of multiple muscles, complex motion involving many degrees-of-freedom can be generated. However, leveraging this concept in robotics typically entails using human data that may be incompatible for the kinematics of the robot. In this paper, our goal is to enable a robot to identify synergies for low-dimensional control using trial-and-error only. We discuss how synergies can be learned through latent space policy search and introduce an extension of the algorithm for the re-use of previously learned synergies for exploration. The application of the algorithm on a bimanual manipulation task for the Baxter robot shows that performance can be increased by reusing learned synergies intra-task when learning to lift objects. But the reuse of synergies between two tasks with different objects did not lead to a significant improvement. Kevin S. Luck, Heni Ben Amor	IROS	2
2017	Robots that anticipate pain: Anticipating physical perturbations from visual cues through deep predictive models abstract To ensure system integrity, robots need to proactively avoid any unwanted physical perturbation that may cause damage to the underlying hardware. In this paper, we investigate a machine learning approach that allows robots to anticipate impending physical perturbations from perceptual cues. In contrast to other approaches that require knowledge about sources of perturbation to be encoded before deployment, our method is based on experiential learning. Robots learn to associate visual cues with subsequent physical perturbations and contacts. In turn, these extracted visual cues are then used to predict potential future perturbations acting on the robot. To this end, we introduce a novel deep network architecture which combines multiple sub-networks for dealing with robot dynamics and perceptual input from the environment. We present a self-supervised approach for training the system that does not require any labeling of training data. Extensive experiments in a human-robot interaction task show that a robot can learn to predict physical contact by a human interaction partner without any prior information or labeling. Indranil Sur, Heni Ben Amor	IROS	2
2017	"Should Robots Feel Pain?" - Towards a Computational Theory of Pain in Autonomous Systems Trevor Richardson, Indranil Sur, Heni Ben Amor	ISRR	3
2016	Sparse Latent Space Policy Search abstract Computational agents often need to learn policies that involve many control variables, e.g., a robot needs to control several joints simultaneously. Learning a policy with a high number of parameters, however, usually requires a large number of training samples. We introduce a reinforcement learning method for sample-efficient policy search that exploits correlations between control variables. Such correlations are particularly frequent in motor skill learning tasks. The introduced method uses Variational Inference to estimate policy parameters, while at the same time uncovering a low-dimensional latent space of controls. Prior knowledge about the task and the structure of the learning agent can be provided by specifying groups of potentially correlated parameters. This information is then used to impose sparsity constraints on the mapping between the high-dimensional space of controls and a lower-dimensional latent space. In experiments with a simulated bi-manual manipulator, the new approach effectively identifies synergies between joints, performs efficient low-dimensional policy search, and outperforms state-of-the-art policy search methods. Kevin S. Luck, Joni Pajarinen, Erik Berger, Ville Kyrki, Heni Ben Amor	AAAI	5
2016	Experience-based torque estimation for an industrial robot abstract Robotic manipulation tasks often require the control of forces and torques exerted on external objects. This paper presents a machine learning approach for estimating forces when no force sensors are present on the robot platform. In the training phase, the robot executes the desired manipulation tasks under controlled conditions with systematically varied parameter sets. All internal sensor data, in the presented case from more than 100 sensors, as well as the force exerted by the robot are recorded. Using Transfer Entropy, a statistical model is learned that identifies the subset of sensors relevant for torque estimation in the given task. At runtime, the model is used to accurately estimate the torques exerted during manipulations of the demonstrated kind. The feasibility of the approach is shown in a setting where a robotic manipulator operates a torque wrench to fasten a screw nut. Torque estimates with an accuracy of well below ±1Nm are achieved. A strength of the presented model is that no prior knowledge of the robot's kinematics, mass distribution or sensor instrumentation is required. Erik Berger, Steve Grehl, David Vogt, Bernhard Jung 0001, Heni Ben Amor	ICRA	5
2016	Estimating perturbations from experience using neural networks and Information Transfer abstract In order to ensure safe operation, robots must be able to reliably detect behavior perturbations that result from unexpected physical interactions with their environment and human co-workers. While some robots provide firmware force sensors that generate rough force estimates, more accurate force measurements are usually achieved with dedicated force-torque sensors. However, such sensors are often heavy, expensive and require an additional power supply. In the case of lightweight manipulators, the already limited payload capabilities may be reduced in a significant way. This paper presents an experience-based approach for accurately estimating external forces being applied to a robot without the need for a force-torque sensor. Using Information Transfer, a subset of sensors relevant to the executed behavior are identified from a larger set of internal sensors. Models mapping robot sensor data to force-torque measurements are learned using a neural network. These models can be used to predict the magnitude and direction of perturbations from affordable, proprioceptive sensors only. Experiments with a UR5 robot show that our method yields force estimates with accuracy comparable to a dedicated force-torque sensor. Moreover, our method yields a substantial improvement in accuracy over force-torque values provided by the robot firmware. Erik Berger, David Vogt, Steve Grehl, Bernhard Jung 0001, Heni Ben Amor	IROS	5
2016	Projecting robot intentions into human environments abstract Trained human co-workers can often easily predict each other's intentions based on prior experience. When collaborating with a robot coworker, however, intentions are hard or impossible to infer. This difficulty of mental introspection makes human-robot collaboration challenging and can lead to dangerous misunderstandings. In this paper, we present a novel, object-aware projection technique that allows robots to visualize task information and intentions on physical objects in the environment. The approach uses modern object tracking methods in order to display information at specific spatial locations taking into account the pose and shape of surrounding objects. As a result, a human co-worker can be informed in a timely manner about the safety of the workspace, the site of next robot manipulation tasks, and next subtasks to perform. A preliminary usability study compares the approach to collaboration approaches based on monitors and printed text. The study indicates that, on average, the user effectiveness and satisfaction is higher with the projection based approach. Rasmus S. Andersen, Ole Madsen, Thomas B. Moeslund, Heni Ben Amor	RO-MAN	4
2015	Occlusion-Aware Object Localization, Segmentation and Pose Estimation abstract We present a learning approach for localization and segmentation of objects in an image in a manner that is robust to partial occlusion. Our algorithm produces a bounding box around the full extent of the object and labels pixels in the interior that belong to the object. Like existing segmentation aware detection approaches, we learn an appearance model of the object and consider regions that do not fit this model as potential occlusions. However, in addition to the established use of pairwise potentials for encouraging local consistency, we use higher order potentials which capture information at the level of im- age segments. We also propose an efficient loss function that targets both localization and segmentation performance. Our algorithm achieves 13.52% segmentation error and 0.81 area under the false-positive per image vs. recall curve on average over the challenging CMU Kitchen Occlusion Dataset. This is a 42.44% decrease in segmentation error and a 16.13% increase in localization performance compared to the state-of-the-art. Finally, we show that the visibility labelling produced by our algorithm can make full 3D pose estimation from a single image robust to occlusion. Samarth Brahmbhatt, Heni Ben Amor, Henrik I. Christensen	BMVC	2
2015	Measuring and Modelling Delays in Robot Manipulators for Temporally Precise Control Using Machine Learning abstract Latencies and delays play an important role in temporally precise robot control. During dynamic tasks in particular, a robot has to account for inherent delays to reach manipulated objects in time. The different types of occurring delays are typically convoluted and thereby hard to measure and separate. In this paper, we present a data-driven methodology for separating and modelling inherent delays during robot control. We show how both actuation and response delays can be modelled using modern machine learning methods. The resulting models can be used to predict the delays as well as the uncertainty of the prediction. Experiments on two widely used robot platforms show significant actuation and response delays in standard control loops. Predictive models can, therefore, be used to reason about expected delays and improve temporal accuracy during control. The approach can easily be used on different robot platforms. Thomas Timm Andersen, Heni Ben Amor, Nils A. Andersen, Ole Ravn	ICMLA	2
2015	Learning multiple collaborative tasks with a mixture of Interaction Primitives abstract Robots that interact with humans must learn to not only adapt to different human partners but also to new interactions. Such a form of learning can be achieved by demonstrations and imitation. A recently introduced method to learn interactions from demonstrations is the framework of Interaction Primitives. While this framework is limited to represent and generalize a single interaction pattern, in practice, interactions between a human and a robot can consist of many different patterns. To overcome this limitation this paper proposes a Mixture of Interaction Primitives to learn multiple interaction patterns from unlabeled demonstrations. Specifically the proposed method uses Gaussian Mixture Models of Interaction Primitives to model nonlinear correlations between the movements of the different agents. We validate our algorithm with two experiments involving interactive tasks between a human and a lightweight robotic arm. In the first, we compare our proposed method with conventional Interaction Primitives in a toy problem scenario where the robot and the human are not linearly correlated. In the second, we present a proof-of-concept experiment where the robot assists a human in assembling a box. Marco Ewerton, Gerhard Neumann, Rudolf Lioutikov, Heni Ben Amor, Jan Peters 0001, Guilherme Maeda	ICRA	4
2015	Exploiting symmetries and extrusions for grasping household objects abstract In this paper we present an approach for creating complete shape representations from a single depth image for robot grasping. We introduce algorithms for completing partial point clouds based on the analysis of symmetry and extrusion patterns in observed shapes. Identified patterns are used to generate a complete mesh of the object, which is, in turn, used for grasp planning. The approach allows robots to predict the shape of objects and include invisible regions into the grasp planning step. We show that the identification of shape patterns, such as extrusions, can be used for fast generation and optimization of grasps. Finally, we present experiments performed with our humanoid robot executing pick-up tasks based on single depth images and discuss the applications and shortcomings of our approach. Ana C. Huamán Quispe, Benoit Milville, Can Erdogan, Mike Stilman, Henrik I. Christensen, Heni Ben Amor	ICRA	7
2015	A Taxonomy of Benchmark Tasks for Robot Manipulation Ana C. Huamán Quispe, Heni Ben Amor, Henrik I. Christensen	ISRR (1)	2
2014	Interaction primitives for human-robot cooperation tasks abstract To engage in cooperative activities with human partners, robots have to possess basic interactive abilities and skills. However, programming such interactive skills is a challenging task, as each interaction partner can have different timing or an alternative way of executing movements. In this paper, we propose to learn interaction skills by observing how two humans engage in a similar task. To this end, we introduce a new representation called Interaction Primitives. Interaction primitives build on the framework of dynamic motor primitives (DMPs) by maintaining a distribution over the parameters of the DMP. With this distribution, we can learn the inherent correlations of cooperative activities which allow us to infer the behavior of the partner and to participate in the cooperation. We will provide algorithms for synchronizing and adapting the behavior of humans and robots during joint physical activities. Heni Ben Amor, Gerhard Neumann, Sanket Kamthe, Oliver Kroemer, Jan Peters 0001	ICRA	1
2014	Latent space policy search for robotics abstract Learning motor skills for robots is a hard task. In particular, a high number of degrees-of-freedom in the robot can pose serious challenges to existing reinforcement learning methods, since it leads to a high-dimensional search space. However, complex robots are often intrinsically redundant systems and, therefore, can be controlled using a latent manifold of much smaller dimensionality. In this paper, we present a novel policy search method that performs efficient reinforcement learning by uncovering the low-dimensional latent space of actuator redundancies. In contrast to previous attempts at combining reinforcement learning and dimensionality reduction, our approach does not perform dimensionality reduction as a preprocessing step but naturally combines it with policy search. Our evaluations show that the new approach outperforms existing algorithms for learning motor skills with high-dimensional robots. Kevin S. Luck, Gerhard Neumann, Erik Berger, Jan Peters 0001, Heni Ben Amor	IROS	5
2014	A Data-Driven Method for Real-Time Character Animation in Human-Agent Interaction David Vogt, Steve Grehl, Erik Berger, Heni Ben Amor, Bernhard Jung 0001	IVA	4
2014	Dynamic Mode Decomposition for perturbation estimation in human robot interaction abstract In many settings, e.g. physical human-robot interaction, robotic behavior must be made robust against more or less spontaneous application of external forces. Typically, this problem is tackled by means of special purpose force sensors which are, however, not available on many robotic platforms. In contrast, we propose a machine learning approach suitable for more common, although often noisy sensors. This machine learning approach makes use of Dynamic Mode Decomposition (DMD) which is able to extract the dynamics of a nonlinear system. It is therefore well suited to separate noise from regular oscillations in sensor readings during cyclic robot movements under different behavior configurations. We demonstrate the feasibility of our approach with an example where physical forces are exerted on a humanoid robot during walking. In a training phase, a snapshot based DMD model for behavior specific parameter configurations is learned. During task execution the robot must detect and estimate the external forces exerted by a human interaction partner. We compare the DMD-based approach to other interpolation schemes and show that the former outperforms the latter particularly in the presence of sensor noise. We conclude that DMD which has so far been mostly used in other fields of science, particularly fluid mechanics, is also a highly promising method for robotics. Erik Berger, Mark Sastuba, David Vogt, Bernhard Jung 0001, Heni Ben Amor	RO-MAN	5
2013	Learning responsive robot behavior by imitation abstract In this paper we present a new approach for learning responsive robot behavior by imitation of human interaction partners. Extending previous work on robot imitation learning, that has so far mostly concentrated on learning from demonstrations by a single actor, we simultaneously record the movements of two humans engaged in on-going interaction tasks and learn compact models of the interaction. Extracted interaction models can thereafter be used by a robot to engage in a similar interaction with a human partner. We present two algorithms for deriving interaction models from motion capture data as well as experimental results on a humanoid robot. Heni Ben Amor, David Vogt, Marco Ewerton, Erik Berger, Bernhard Jung 0001, Jan Peters 0001	IROS	1
2012	Generalization of human grasping for multi-fingered robot hands abstract Multi-fingered robot grasping is a challenging problem that is difficult to tackle using hand-coded programs. In this paper we present an imitation learning approach for learning and generalizing grasping skills based on human demonstrations. To this end, we split the task of synthesizing a grasping motion into three parts: (1) learning efficient grasp representations from human demonstrations, (2) warping contact points onto new objects, and (3) optimizing and executing the reach-and-grasp movements. We learn low-dimensional latent grasp spaces for different grasp types, which form the basis for a novel extension to dynamic motor primitives. These latent-space dynamic motor primitives are used to synthesize entire reach-and-grasp movements. We evaluated our method on a real humanoid robot. The results of the experiment demonstrate the robustness and versatility of our approach. Heni Ben Amor, Oliver Kroemer, Ulrich Hillenbrand, Gerhard Neumann, Jan Peters 0001	IROS	1
2012	Maximally informative interaction learning for scene exploration abstract Creating robots that can act autonomously in dynamic, unstructured environments is a major challenge. In such environments, learning to recognize and manipulate novel objects is an important capability. A truly autonomous robot acquires knowledge through interaction with its environment without using heuristics or prior information encoding human domain insights. Static images often provide insufficient information for inferring the relevant properties of the objects in a scene. Hence, a robot needs to explore these objects by interacting with them. However, there may be many exploratory actions possible, and a large portion of these actions may be non-informative. To learn quickly and efficiently, a robot must select actions that are expected to have the most informative outcomes. In the proposed bottom-up approach, the robot achieves this goal by quantifying the expected informativeness of its own actions. We use this approach to segment a scene into its constituent objects as a first step in learning the properties and affordances of objects. Evaluations showed that the proposed information-theoretic approach allows a robot to efficiently infer the composite structure of its environment. Herke van Hoof, Oliver Kroemer, Heni Ben Amor, Jan Peters 0001	IROS	3
2009	Physical interaction learning: Behavior adaptation in cooperative human-robot tasks involving physical contact abstract In order for humans and robots to engage in direct physical interaction several requirements have to be met. Among others, robots need to be able to adapt their behavior in order to facilitate the interaction with a human partner. This can be achieved using machine learning techniques. However, most machine learning scenarios to-date do not address the question of how learning can be achieved for tightly coupled, physical touch interactions between the learning agent and a human partner. This paper presents an example for such human in-the-loop learning scenarios and proposes a computationally cheap learning algorithm for this purpose. The efficiency of this method is evaluated in an experiment, where human care givers help an android robot to stand up. Shuhei Ikemoto, Heni Ben Amor, Takashi Minato, Hiroshi Ishiguro, Bernhard Jung 0001	RO-MAN	2
2008	Identifying Motion Capture Tracking Markers with Self-Organizing Maps abstract Motion capture (MoCap) describes methods and technologies for the detection and measurement of human motion in all its intricacies. Most systems use markers to track points on a body. Especially with natural human motion data captured with passive systems (to not hinder the participant) deficiencies like low accuracy of tracked points or even occluded markers might occur. Additionally, such MoCap data is often unlabeled. In consequence, the system does not provide information about which body landmarks the points belong to. Self-organizing neural networks, especially self- organizing maps (SOMs), are capable of dealing with such problems. This work describes a method to model, initialize and train such SOMs to track and label potentially noisy motion capture data. Matthias Weber 0004, Heni Ben Amor	VR	2
2008	Grasp synthesis from low-dimensional probabilistic grasp models abstract Abstract We propose a novel data‐driven animation method for the synthesis of natural looking human grasping. Motion data captured from human grasp actions is used to train a probabilistic model of the human grasp space. This model greatly reduces the high number of degrees of freedom of the human hand to a few dimensions in a continuous grasp space. The low dimensionality of the grasp space in turn allows for efficient optimization when synthesizing grasps for arbitrary objects. The method requires only a short training phase with no need for preprocessing of graphical objects for which grasps are to be synthesized. Copyright © 2008 John Wiley & Sons, Ltd. Heni Ben Amor, Guido Heumer, Bernhard Jung 0001, Arnd Vitzthum	Comput. Animat. Virtual Worlds	1
2007	Grasp Recognition with Uncalibrated Data Gloves - A Comparison of Classification Methods abstract This paper presents a comparison of various classification methods for the problem of recognizing grasp types involved in object manipulations performed with a data glove. Conventional wisdom holds that data gloves need calibration in order to obtain accurate results. However, calibration is a time-consuming process, inherently user-specific, and its results are often not perfect. In contrast, the present study aims at evaluating recognition methods that do not require prior calibration of the data glove, by using raw sensor readings as input features and mapping them directly to different categories of hand shapes. An experiment was carried out, where test persons wearing a data glove had to grasp physical objects of different shapes corresponding to the various grasp types of the Schlesinger taxonomy. The collected data was analyzed with 28 classifiers including different types of neural networks, decision trees, Bayes nets, and lazy learners. Each classifier was analyzed in six different settings, representing various application scenarios with differing generalization demands. The results of this work are twofold: (1) We show that a reasonably well to highly reliable recognition of grasp types can be achieved - depending on whether or not the glove user is among those training the classifier - even with uncalibrated data gloves. (2) We identify the best performing classification methods for recognition of various grasp types. To conclude, cumbersome calibration processes before productive usage of data gloves can be spared in many situations. Guido Heumer, Heni Ben Amor, Matthias Weber 0004, Bernhard Jung 0001	VR	2
2006	From motion capture to action capture: a review of imitation learning techniques and their application to VR-based character animation abstract We present a novel method for virtual character animation that we call action capture. In this approach, virtual characters learn to imitate the actions of Virtual Reality (VR) users by tracking not only the users' movements but also their interactions with scene objects.Action capture builds on conventional motion capture but differs from it in that higher-level action representations are transferred rather than low-level motion data. As an advantage, the learned actions can often be naturally applied to varying situations, thus avoiding retargetting problems of motion capture. The idea of action capture is inspired by human imitation learning; related methods have been investigated for a longer time in robotics. The paper reviews the relevant literature in these areas before framing the concept of action capture in the context of VR-based character animation. We also present an example in which the actions of a VR user are transferred to a virtual worked. Bernhard Jung 0001, Heni Ben Amor, Guido Heumer, Matthias Weber 0004	VRST	2
2005	Intelligent exploration for genetic algorithms: using self-organizing maps in evolutionary computation Heni Ben Amor, Achim Rettinger	GECCO	1