Students with disabilities face a variety of challenges in computer science education including those related to inaccessible curriculum, instruction and tools. In addition, few computer science classes teach students to understand accessibility and design accessible technology. This BOF will bring together individuals who are interested in increasing the accessibility of computing education as well as those interested in teaching about accessibility. Participants will share strategies to help each other do a better job of addressing accessibility in our classes and research projects. Resources related to accessible tools and instruction, universal design of learning, opportunities for students, teaching accessibility, and more will be shared.

Several countries legally require computing programs to ensure their graduates can develop accessible computing solutions. Computing Science Curricula 2023 (CS2023), developed by a joint task force from ACM, IEEE-Computer Society, and AAAI, likewise mandates coverage of accessibility and provides guidance on how to integrate accessibility into computing curricula. Yet the accreditation criteria used by ABET's Computing Accreditation Commission (CAC) -- which accredits more than 650 computing programs in 25 countries -- do not currently require graduates from these programs to know about computing accessibility. Adding accessibility to the CAC accreditation criteria would be transformative in closing the growing gap of accessibility skills in the computing industry, and ensuring new computing technology is accessible to all.

Collaboration and community building are crucial for equitable participation in research, especially in disability and accessibility research fields.Engaging with communities of disabled individuals helps understand their unique challenges, expectations, and needs, which should be reflected in assistive technology design.However, a gap exists between research in resource-rich societies and the practices of low-resource settings, where socio-economic and infrastructural constraints are often overlooked.Conversely, the innovation and sustainable solutions in these settings are often

Millions of individuals worldwide experience signif-icant disability, often relying on caregivers for activities of daily living such as eating, bathing, ambulating, and dressing. This reliance on caregivers can negatively impact their mental health and can place a considerable physical workload on caregivers. Physical robot caregiving has emerged as a promising solution to this challenge, with recent years seeing a surge of research interest in developing physically assistive robots for people with disabilities. This workshop focuses on bringing together researchers, end-users, caregivers, and healthcare professionals to discuss existing research on physical caregiving robots, identify gaps, foster collaborations, and chart future research directions.

Millions of people cannot eat independently due to a disability, and caregiver-assisted meals can make them feel self-conscious, pressured, or burdensome. Robot-assisted feeding promises to empower people with motor impairments to feed themselves. However, current research typically examines specific robotic system subcomponents and evaluates them in controlled lab settings. This leaves a gap in developing and evaluating an end-to-end system that can feed entire meals in out-of-lab settings. We present one such system, which we developed collaboratively with two community researchers (CRs) with motor-impairments. The key challenge of developing a robot feeding system for out-of-lab use is the varied off-nominal scenarios that inevitably arise. Our key insight is that users can overcome many off-nominals, provided customizability and control over the system. Our system improves upon the state-of-the-art with: (1) a user interface that provides substantial user customizability and control, (2) a bite selection implementation that incorporates users-in-the-loop to generalize across food items, and (3) portable hardware that facilitates system use in diverse environments without inhibiting user mobility. We conduct two studies to evaluate the system. In Study 1, five users with motor impairments and one CR use the system to feed themselves meals of their choice in a cafeteria, office, or conference room. In Study 2, one CR uses the system in his home for five days, feeding himself 10 meals across diverse contexts. We present 3 key lesson learned: (1) spatial contexts are numerous, customizability lets users adapt to them; (2) off-nominals will arise, variable autonomy lets users overcome them; and (3) assistive robots' benefits depend on context. We provide video footage and code on our website.

Humanoid robots are the latest bet of the robotics community in advancing ways robots carry out a large variety of tasks that generate profit or increase quality of life for people. While their capabilities might extend to assistive care tasks, such as feeding, dressing, or household tasks, it is unclear if people are comfortable with having humanoid robots in their homes assisting with those tasks. In this paper we explore people's attitudes towards assistive humanoid robots in the the home. We present two questionnaire studies, with 76 total participants, in which people are shown imaginary images of humanoid robots performing assistance tasks in the home, along with special purpose robot alternatives. Participants are asked to rate and compare robots in the context of eight different tasks and share their reasoning. The second study also shows participants pictures of real humanoid robot both without any context and in the context of in-home assistance tasks, and asks their opinions about these robots. Our findings indicate that people prefer special purpose robots over humanoids in most cases and their preferences vary by task. Although people think that humanoids are acceptable for assistance with some tasks, they express concerns about having them in their homes.

Roboticists often design with the assumption that assistive robots should be fully autonomous. However, it remains unclear whether users prefer highly autonomous robots, as prior work in assistive robotics suggests otherwise. High robot autonomy can reduce the user's sense of agency, which represents feeling in control of one's environment. How much control do users, in fact, want over the actions of robots used for in-home assistance? We investigate how robot autonomy levels affect users' sense of agency and the autonomy level they prefer in contexts with varying risks. Our study asked participants to rate their sense of agency as robot users across four distinct autonomy levels and ranked their robot preferences with respect to various household tasks. Our findings revealed that participants' sense of agency was primarily influenced by two factors: (1) whether the robot acts autonomously, and (2) whether a third party is involved in the robot's programming or operation. Notably, an end-user programmed robot highly preserved users' sense of agency, even though it acts autonomously. However, in high-risk settings, e.g., preparing a snack for a child with allergies, they preferred robots that prioritized their control significantly more. Additional contextual factors, such as trust in a third party operator, also shaped their preferences.

Students with disabilities face a variety of challenges in computer science education including those related to stigma around disability, inaccessible curriculum, instruction and tools, disability disclosure, and a lack of mentors. In addition, few computer science classes teach students to understand accessibility and design accessible technology. This BOF will bring together individuals who are interested in increasing the representation of people with disabilities in computing and improving their success as well as those interested in teaching about accessibility. Participants will share strategies to help each other do a better job of addressing disability inclusion and accessibility in our classes and research projects. Resources related to accessible tools and instruction, universal design of learning, opportunities for students, teaching accessibility, and more will be shared.

Fluid 6DOF teleoperation of robot manipulators enables telemanipulation where autonomy is not possible, facilitates the collection of demonstration data, and aids routine robotics development. Amongst 6DOF input devices, 3D mice stand apart for their ergonomic design and low cost, but their sensitivity and users' relative inexperience with them require special design considerations. We contribute a web software package that makes integrating 3D mice in robot manipulation interfaces easy. The package consists of configurable input signal processing schemes that can make the device more forgiving by, for instance, rejecting small inputs or emphasizing a dominant axis, and an interactive visual representation of the device's 6DOF twist input, which helps with operator familiarization and provides a visual aide during teleoperation. We provide a demonstration interface illustrating a typical integration with a ROS/ROS2 robot system and give usage advice based on our research experience.

Modern warehouses process millions of unique objects which are often stored in densely packed containers. To automate tasks in this environment, a robot must be able to pick diverse objects from highly cluttered scenes. Real-world learning is a promising approach, but executing picks in the real world is time-consuming, can induce costly failures, and often requires extensive human intervention, which causes operational burden and limits the scope of data collection and deployments. In this work, we leverage interactive probes to visually evaluate grasps in clutter without fully executing picks, a capability we refer to as Interactive Visual Failure Prediction (IVFP). This enables autonomous verification of grasps during execution to avoid costly downstream failures as well as autonomous reward assignment, providing supervision to continuously shape and improve grasping behavior as the robot gathers experience in the real world, without constantly requiring human intervention. Through experiments on a Stretch RE1 robot, we study the effect that IVFP has on performance - both in terms of effective data throughput and success rate, and show that this approach leads to grasping policies that outperform policies trained with human supervision alone, while requiring significantly less human intervention. Code, datasets, and videos available at https://robo-ivfp.github.io

The performance of prediction-based assistance for robot teleoperation degrades in unseen or goal-rich environments due to incorrect or quickly-changing intent inferences. Poor predictions can confuse operators or cause them to change their control input to implicitly signal their goal. We present a new assistance interface for robotic manipulation where an operator can explicitly communicate a manipulation goal by pointing the end-effector. The pointing target specifies a region for local pose generation and optimization, providing interactive control over grasp and placement pose candidates. We evaluate this explicit pointing interface against an implicit inference-based assistance scheme and an unassisted control condition in a within-subjects user study (N=20), where participants teleoperate a simulated robot to complete a multi-step singulation and stacking task in cluttered environments. We find that operators prefer the explicit interface, experience fewer pick failures and report lower cognitive workload. Our code is available at: github.com/NVlabs/fast-explicit-teleop.

Programming by Demonstration (PbD) is an intuitive technique for programming robot manipulation skills by demonstrating the desired behavior. However, most existing approaches either require extensive demonstrations or fail to generalize beyond their initial demonstration conditions. We introduce Diffusion-PbD, a novel approach to PbD that enables users to synthesize generalizable robot manipulation skills from a single demonstration by utilizing the representations captured by pre-trained visual foundation models. At demonstration time, hand and object detection priors are used to extract waypoints from the human demonstrations anchored to reference points in the scene. At execution time, features from pre-trained diffusion models are leveraged to identify corresponding reference points in new observations. We validate this approach through a series of real-world robot experiments, showing that Diffusion-PbD is applicable to a wide range of manipulation tasks and has strong ability to generalize to unseen objects, camera viewpoints, and scenes. Code and supplementary videos can be found at https://diffusion-pbd.github.io

Mobile manipulator robots, which can move around and physically interact with their environments, can empower people with motor limitations to independently carry out many activities of daily living. While many interfaces have been developed for tele-operating complex robots, most of them are not accessible to people with severe motor limitations. Further, most interfaces are rigid with limited configurations and are not readily available to download and use. To address these barriers, we developed AccessTeleopKit: an open-source toolkit for creating custom and accessible robot tele-operation interfaces based on cursor-and-click input for the Stretch 3 mobile-manipulator. With AccessTeleopKit users can add, remove, and rearrange components such as buttons and camera views, and select between a variety of control modes. We describe the participatory and iterative design process that led to the current implementation of AccessTeleopKit, involving three long-term deployments of the robot in the home of a quadriplegic user. We demonstrate how AccessTeleopKit allowed the user to create different interfaces for different tasks and the diversity of tasks it allowed the user to carry out. We also present two studies involving six additional users with severe motor limitations, demonstrating the power of AccessTeleopKit in creating custom interfaces for different user needs and preferences.

Social dining, i.e., eating with/in company, is replete with meaning and cultural significance. Unfortunately, for the 1.8 million Americans with motor impairments who cannot eat without assistance, challenges restrict them from enjoying this pleasant social ritual. In this work, we identify the needs of participants with motor impairments during social dining and how robot-assisted feeding can address them. Using speculative videos that show robot behaviors within a social dining context, we interviewed participants to understand their preferences. Following a community-based participatory research method, we worked with a community researcher with motor impairments throughout this study. We contribute (a) insights into how a robot can help overcome challenges in social dining, (b) design principles for creating robot-assisted feeding systems, (c) and an implementation guide for future research in this area. Our key finding is that robots' unique assistive qualities can address challenges people with motor impairments face during social dining, promoting empowerment and belonging.

Service robots for personal use in the home and the workplace require end-user development solutions for swiftly scripting robot tasks as the need arises. Many existing solutions preserve ease, efficiency, and convenience through simple programming interfaces or by restricting task complexity. Others facilitate meticulous task design but often do so at the expense of simplicity and efficiency. There is a need for robot programming solutions that reconcile the complexity of robotics with the on-the-fly goals of end-user development. In response to this need, we present a novel, multimodal, and on-the-fly development system, Tabula. Inspired by a formative design study with a prototype, Tabula leverages a combination of spoken language for specifying the core of a robot task and sketching for contextualizing the core. The result is that developers can script partial, sloppy versions of robot programs to be completed and refined by a program synthesizer. Lastly, we demonstrate our anticipated use cases of Tabula via a set of application scenarios.

Mobile manipulator platforms, like the Stretch RE1 robot, make the promise of in-home robotic assistance feasible. For people with severe physical limitations, like those with quadriplegia, the ability to tele-operate these robots themselves means that they can perform physical tasks they cannot otherwise do themselves, thereby increasing their level of independence. In order for users with physical limitations to operate these robots, their interfaces must be accessible and cater to the specific needs of all users. As physical limitations vary amongst users, it is difficult to make a single interface that will accommodate all users. Instead, such interfaces should be customizable to each individual user. In this paper we explore the value of customization of a browser-based interface for tele-operating the Stretch RE1 robot. More specifically, we evaluate the usability and effectiveness of a customized interface in comparison to the default interface configurations from prior work. We present a user study involving participants with motor impairments $(\mathrm{N}=10)$ and without motor impairments, who could serve as a caregiver, $(\mathrm{N}=13)$ that use the robot to perform mobile manipulation tasks in a real kitchen environment. Our study demonstrates that no single interface configuration satisfies all users’ needs and preferences. Users perform better when using the customized interface for navigation, but not for manipulation due to higher complexity of learning to manipulate through the robot. All participants are able to use the robot to complete all tasks, and participants with motor impairments believe that having the robot in their home would make them more independent.

The social robotics market is appealing yet challenging. Though social robots are built few remain on the market for long. Many reasons account for their short lifespan with costs and context-specificity ranking high amount them. In this work, we designed, fabricated, and developed FLEXI, a social robot embodiment kit that enabled unlimited customization, making it applicable for a broad range of use cases. The hardware and software of FLEXI were entirely developed by this research team from scratch. FLEXI includes a rich set of materials and attachment pieces to allow for a diverse range of hardware customizations that ensure the embodiment is appropriate for specific customer/researcher projects. It also includes an open-source end-user programming interface to lower the barrier of robotics access to interdisciplinary teams that populate the field of Human-Robot Interaction. We present an iterative development of this cost-effective kit through the lenses of case studies, conceptual research, and soft deployment of FLEXI in three application scenarios: community-support, mental health, and education. Additionally, we provide in open-access the full list of materials and a tutorial to fabricate FLEXI, making it accessible to any maker space, research lab, or workshop space interested in working with or learning about social robots.

One of the core challenges in creating interactive behaviors for social robots is testing. Programs implementing the interactive behaviors require real humans to test and this requirement makes testing of the programs extremely expensive. To address this problem, human-robot interaction researchers in the past proposed using human simulators. However, human simulators are tedious to set up and context-dependent and therefore are not widely used in practice. We propose a program synthesis approach to building human simulators for the purpose of testing interactive robot programs. Our key ideas are (1) rep-resenting human simulators as probabilistic functional reactive programming programs and (2) using probabilistic inference for synthesizing human simulator programs. Programmers then will be able to build human simulators by providing interaction traces between a robot and a human or two humans which they can later use to test interactive robot programs and improve or tweak as needed.

Social robots have been used to support mental health. In this work, we explored their potential as community-based tools. Visualizing mood data patterns of a community with a social robot might help the community raise awareness about the emotions people feel and affecting factors from life events. This could potentially lead to adaptation of suitable coping skills enhancing the sense of belonging and support among community members. We present preliminary findings and ongoing plans for this human-robot interaction (HRI) research work on data visualizations supporting community mental health. In a two-day study, twelve participants recruited from a university community engaged with a robot displaying mood data. Given the feedback from the study, we improved the data visualization in the robot to increase accessibility, universality, and usefulness of such visualizations. In the future, we plan on conducting studies with this improved version and deploying a social robot for a community setting.

It is difficult to run long-term in-the-wild studies with mobile robots. This is partly because the robots we, as human-robot interaction (HRI) researchers, are interested in deploying prioritize expressivity over navigational capabilities, and making those robots autonomous is often not the focus of our research. One way to address these difficulties is with the Wizard of Oz (WoZ) methodology, where a researcher teleop-erates the robot during its deployment. However, the constant attention required for teleoperation limits the duration of WoZ deployments, which in-turn reduces the amount of in-the-wild data we are able to collect. Our key insight is that several types of in-the-wild mobile robot studies can be run without autonomous navigation, using wandering instead. In this paper we present and share code for our wandering robot system, which enabled Kuri, an expressive robot with limited sensor and computational capabilities, to traverse the hallways of a$28,000 \text{ ft}^{2}$floor for four days. Our system relies on informed direction selection to avoid obstacles and traverse the space, and periodic human help to charge. After presenting the outcomes from the four-day deployment, we then discuss the benefits of deploying a wandering robot, explore the types of in-the-wild studies that can be run with wandering robots, and share pointers for enabling other robots to wander. Our goal is to add wandering to the toolbox of navigation approaches HRI researchers use, particularly to run in-the-wild deployments with mobile robots.

We introduce a new simulation benchmark “Han-doverSim” for human-to-robot object handovers. To simulate the giver's motion, we leverage a recent motion capture dataset of hand grasping of objects. We create training and evaluation environments for the receiver with standardized protocols and metrics. We analyze the performance of a set of baselines and show a correlation with a real-world evaluation.11Code is open sourced at https://handover-sim.github.io.

Human-robot handover is a fundamental yet challenging task in human-robot interaction and collaboration. Recently, remarkable progressions have been made in human-to-robot handovers of unknown objects by using learning-based grasp generators. However, how to responsively generate smooth motions to take an object from a human is still an open question. Specifically, planning motions that take human comfort into account is not a part of the human-robot handover process in most prior works. In this paper, we propose to generate smooth motions via an efficient model-predictive control (MPC) framework that integrates perception and complex domain-specific constraints into the optimization problem. We introduce a learning-based grasp reachability model to select candidate grasps which maximize the robot's manipulability, giving it more freedom to satisfy these constraints. Finally, we integrate a neural net force/torque classifier that detects contact events from noisy data. We conducted human-to-robot handover experiments on a diverse set of objects with several users ($N=4$) and performed a systematic evaluation of each module. The study shows that the users preferred our MPC approach over the baseline system by a large margin.

Adolescents isolated at home during the COVID19 pandemic lockdown are more likely to feel lonely and in need of social connection. Social robots may provide a much needed social interaction without the risk of contracting an infection. In this paper, we detail our co-design process used to engage adolescents in the design of a social robot prototype intended to broadly support their mental health. Data gathered from our four week design study of nine remote sessions and interviews with 16 adolescents suggested the following design requirements for a home robot: (1) be able to enact a set of roles including a coach, companion, and confidant; (2) amplify human-to-human connection by supporting peer relationships; (3) account for data privacy and device ownership. Design materials are available in open-access, contributing to best practices for the field of Human-Robot Interaction.

We present FLEX-SDK: an open-source software development kit that allows creating a social robot from two simple tablet screens. FLEX-SDK involves tools for designing the robot face and its facial expressions, creating screens for input/output interactions, controlling the robot through a Wizard-of-Oz interface, and scripting autonomous interactions through a simple text-based programming interface. We demonstrate how this system can be used to replicate an interaction study and we present nine case studies involving controlled experiments, observational studies, participatory design sessions, and outreach activities in which our tools were used by researchers and participants to create and interact with social robots. We discuss common observations and lessons learned from these case studies. Our work demonstrates the potential of FLEX-SDK to lower the barrier to entry for Human-Robot Interaction research.

Remote teleoperation is an important robot control method when they cannot operate fully autonomously. Yet, teleoperation presents challenges to effective and full robot utilization: controls are cumbersome, inefficient, and the teleoperator needs to actively attend to the robot and its environment. Inspired by end-user programming, we propose a new interaction paradigm to support robot teleoperation for combinations of repetitive and complex movements. We introduce Mimic, a system that allows teleoperators to demonstrate and save robot trajectories as templates, and re-use them to execute the same action in new situations. Templates can be re-used through (1) macros—parametrized templates assigned to and activated by buttons on the controller, and (2) programs—sequences of parametrized templates that operate autonomously. A user study in a simulated environment showed that after initial set up time, participants completed manipulation tasks faster and more easily compared to traditional direct control.

Human-robot interaction designers and developers navigate a complex design space, which creates a need for tools that support intuitive design processes and harness the programming capacity of state-of-the-art authoring environments. We introduce Figaro, an expressive tabletop authoring environment for mobile robots, inspired by shadow puppetry, that provides designers with a natural, situated representation of human-robot interactions while exploiting the intuitiveness of tabletop and tangible programming interfaces. On the tabletop, Figaro projects a representation of an environment. Users demonstrate sequences of behaviors, or scenes, of an interaction by manipulating instrumented figurines that represent the robot and the human. During a scene, Figaro records the movement of figurines on the tabletop and narrations uttered by users. Subsequently, Figaro employs real-time program synthesis to assemble a complete robot program from all scenes provided. Through a user study, we demonstrate the ability of Figaro to support design exploration and development for human-robot interaction.

Human-robot object handovers have been an actively studied area of robotics over the past decade; however, very few techniques and systems have addressed the challenge of handing over diverse objects with arbitrary appearance, size, shape, and deformability. In this paper, we present a vision-based system that enables reactive human-to-robot handovers of unknown objects. Our approach combines closed-loop motion planning with real-time, temporally consistent grasp generation to ensure reactivity and motion smoothness. Our system is robust to different object positions and orientations, and can grasp both rigid and non-rigid objects. We demonstrate the generalizability, usability, and robustness of our approach on a novel benchmark set of 26 diverse household objects, a user study with six participants handing over a subset of 15 objects, and a systematic evaluation examining different ways of handing objects.

Cursor-based tele-operation interfaces for manipulators can enable widely available and accessible control of robots to make many near term applications possible. However, their efficiency is restricted by the challenge of controlling 6 Degrees-of-Freedom (DoF) with 2D input from the cursor. Existing interfaces make use of different strategies to tackle this challenge, including viewpoint constraints, mode switching, and visual overlays, but it is unclear how these strategies impact the efficiency and accessibility of the interface. In this paper we characterize the design space of cursor-based robot control interfaces and compare alternatives in two user studies. Study 1 (N=216) compares nine alternative interfaces focusing on control of 3 DoFs to understand the differences of the interfaces at the basic level and examine the impact of task parameters on efficiency. Study 2 (N=60) compares a subset of the interfaces integrated into a system that allows full control of a robot manipulator from three orthogonal views. We also present a framework for heuristically evaluating accessibility of these interfaces and discuss the efficiency and accessibility trade-off with recommendations.

New mobile manipulator platforms, like the Hello Robot Stretch, have made the idea of long-term in-home robotic assistance feasible. However, existing autonomous capabilities for such robots in unstructured, highly-varied environments are still not available. Instead, using robots with human tele-operation can have huge immediate impact. For these robots to serve populations that need them the most, their interfaces need to be accessible to people with mobility limitations. In this paper we explore the utility, usability, and accessibility of a tele-operated Stretch robot in the home. We first describe a browser-based interface for controlling the Stretch robot designed with accessibility in mind. We then present findings from a study (N=18) in which participants used the interface to remotely control the robot to perform realistic tasks in a kitchen, demonstrating the feasibility of tele-operated assistance and revealing challenges and opportunities. Next, we present a study with individuals with mobility limitations (N=3) identifying additional accessibility requirements for the interface. Participants in both studies agree to the utility of the robot despite current limitations.

Researchers have proposed models of curiosity as a means to drive robots to learn and adapt to their environments. While these models balance goal- and exploration-oriented actions in a mathematically principled manor, it is not understood how users perceive a robot that pursues off-task actions. Motivated by a model of curiosity based on intrinsic rewards, we conducted three online video-surveys with a total of 264 participants, evaluating a variety of curious behaviors. Our results indicate that a robot's off-task actions are perceived as expressions of curiosity, but that these actions lead to a negative impact on perceptions of the robot's competence. When the robot explains or acknowledges its deviation from the primary task, this can partially mitigate the negative effects of off-task actions.

A robot-assisted feeding system can potentially help a user with upper-body mobility impairments eat independently. However, autonomous assistance in the real world is challenging because of varying user preferences, impairment constraints, and possibility of errors in uncertain and unstructured environments. An autonomous robot-assisted feeding system needs to decide the appropriate strategy to acquire a bite of hard-to-model deformable food items, the right time to bring the bite close to the mouth, and the appropriate strategy to transfer the bite easily. Our key insight is that a system should be designed based on a user's preference about these various challenging aspects of the task. In this work, we explore user preferences for different modes of autonomy given perceived error risks and also analyze the effect of input modalities on technology acceptance. We found that more autonomy is not always better, as participants did not have a preference to use a robot with partial autonomy over a robot with low autonomy. In addition, participants' user interface preference changes from voice control during individual dining to web-based during social dining. Finally, we found differences on average ratings when grouping the participants based on their mobility limitations (lower vs. higher) that suggests that ratings from participants with lower mobility limitations are correlated with higher expectations of robot performance.

Enabling diverse users to program robots for different applications is critical for robots to be widely adopted. Most of the new collaborative robot manipulators come with intuitive programming interfaces that allow novice users to compose robot programs and tune their parameters. However, parameters like motion speeds or exerted forces cannot be easily demonstrated and often require manual tuning, resulting in a tedious trial-and-error process. To address this problem, we formulate tuning of one-dimensional parameters as an Active Learning problem where the learner iteratively refines its estimate of the feasible range of parameter values, by selecting informative queries. By executing the parametrized actions, the learner gathers the user's feedback, in the form of directional answers ("higher,'' "lower,'' or "fine''), and integrates it in the estimate. We propose an Active Learning approach based on Expected Divergence Maximization for this setting and compare it against two baselines with synthetic data. We further compare the approaches on a real-robot dataset obtained from programs written with a simple Domain-Specific Language for a robot arm and manually tuned by expert users (N=8) to perform four manipulation tasks. We evaluate the effectiveness and usability of our interactive tuning approach against manual tuning with a user study where novice users (N=8) tuned parameters of a human-robot hand-over program.

Transfer of objects between humans and robots is a critical capability for collaborative robots. Although there has been a recent surge of interest in human-robot handovers, most prior research focus on robot-to-human handovers. Further, work on the equally critical human-to-robot handovers often assumes humans can place the object in the robot's gripper. In this paper, we propose an approach for human-to-robot handovers in which the robot meets the human halfway, by classifying the human's grasp of the object and quickly planning a trajectory accordingly to take the object from the human's hand according to their intent. To do this, we collect a human grasp dataset which covers typical ways of holding objects with various hand shapes and poses, and learn a deep model on this dataset to classify the hand grasps into one of these categories. We present a planning and execution approach that takes the object from the human hand according to the detected grasp and hand position, and replans as necessary when the handover is interrupted. Through a systematic evaluation, we demonstrate that our system results in more fluent handovers versus two baselines. We also present findings from a user study (N = 9) demonstrating the effectiveness and usability of our approach with naive users in different scenarios. More information can be found at http://wyang.me/handovers.

Concurrency makes robot programming challenging even for professional programmers, yet it is essential for rich, interactive social robot behaviors. Visual programming aims to lower the barrier for robot programming but does not support rich concurrent behavior for meaningful robotics applications. In this paper, we explore extensions to block-based visual languages to enable programming of concurrent behavior with (1) asynchronous procedure calls, which encourage imperative programming, (2) callbacks, which encourage event-driven programming, and (3) promise, which also encourages imperative programming by providing event synchronization utilities. We compare these approaches through a systematic analysis of social robot programs with representative concurrency patterns, as well as a user study (N=23) in which participants authored such programs. Our work identifies characteristic differences between these approaches and demonstrates that the promise-based concurrency interface enables more concise programs with fewer errors.

Design requirements can be gathered through a variety of ways; however, engaging teen audiences in design process can be challenging. We present a novel method for engaging teens in design through a social robot design challenge. Groups of teens participated in the challenge to prototype a social robot that would live in their high school and help address stress, a persistent and pervasive problem for this age group. In this paper, we present our methods and share preliminary findings.

Robot-assisted feeding (RAF) systems offer enormous potential benefits to community-centered care-giving environments. However, developers of RAF technologies often focus on evaluating their standard transactional functionality, omitting the impact of such technologies in contexts that extend past the interaction of the robot and food receiver. RAF technologies have complex social, cultural and self-identity implications, since a "meal" extends well beyond the simple provisioning of nourishment. To better understand these implications we conducted a contextual inquiry in an assisted-living community with five potential care recipients and five caregivers, as well as interviews with fifteen domain experts including occupational therapists and feeding specialists. Based on our findings from these studies, we developed a new framework for RAF technologies that formulates this vital task as a community-centered relational service. We then use this framework to qualitatively and quantitatively assess three existing feeding systems and identify areas of improvement. Our work reveals new insights about stakeholders of RAF technologies and provides a roadmap for technology developers to better serve the needs of these stakeholders.

The ability to reference objects in the environment is a key communication skill that robots need for complex, task-oriented human-robot collaborations. In this paper we explore the use of projections, which are a powerful communication channel for robot-to-human information transfer as they allow for situated, instantaneous, and parallelized visual referencing. We focus on the question of what makes a good projection for referencing a target object. To that end, we mathematically formulatelegibility of projections intended to reference an object, and propose alternative arrow-object match functions for optimally computing the placement of an arrow to indicate a target object in a cluttered scene. We implement our approach on a PR2 robot with a head-mounted projector. Through an online (48 participants) and an in-person (12 participants) user study we validate the effectiveness of our approach, identify the types of scenes where projections may fail, and characterize the differences between alternative match functions.

Programming by Demonstration (PbD) lets users with little technical background program a wide variety of manipulation tasks for robots, but it should be as intuitive as possible for users while requiring as little time as possible. In this paper, we present a Programming by Demonstration system that synthesizes manipulation programs from a single observed demonstration, allowing users to program new tasks for a robot simply by performing the task once themselves. A human-in-the-loop interface helps users make corrections to the perceptual state as needed. We introduce Object Interaction Programs as a representation of multi-object, bimanual manipulation tasks and present algorithms for extracting programs from observed demonstrations and transferring programs to a robot to perform the task in a new scene. We demonstrate the expressivity and generalizability of our approach through an evaluation on a benchmark of complex tasks.

Understanding people's attitudes towards robots and how those attitudes are affected by exposure to robots is essential to the effective design and development of social robots. Although researchers have been studying attitudes towards robots among adults and even children for more than a decade, little has been explored assessing attitudes among teens-a highly vulnerable population that presents unique opportunities and challenges for social robots. Our work aims to close this gap. In this paper we present findings from several participatory robot interaction and design sessions with 136 teenagers who completed a modified version of the Negative Attitudes Towards Robots Scale (NARS) before participation in a robot interaction. Our data reveal that most teens are 1) highly optimistic about the helpfulness of robots, 2) do not feel nervous talking with a robot, but also 3) do not trust a robot with their data. Ninety teens also completed a post-interaction survey and reported a significant change in the motional attitudes subscale of the NARS. We discuss the implications of our findings on the design of social robots for teens.

Maintaining independence and dignity is a primary goal of successful aging for older adults around the globe. Robots can support this goal in various ways by assisting everyday tasks that become challenging due to aging-related deterioration in physical and mental abilities. While a growing body of research tackles challenges in creating such robots, most work has focused on older adults with high socio-economic status in the developed world. In most cases, the price of these robots alone prohibits their potential use in the developing world. Further, socio-cultural differences in the developing world will limit the usability and chance of adoption of a robot designed based on users in the developed world. Our work aims to close this gap. In this paper we present findings from the user-centered design and development process of a low-cost assistive robot for older adults in the developing world named SHEBA, which is a Bengali term for care. We first interviewed 37 older adults and 21 caregivers in assisted and independent living settings in Dhaka, Bangladesh to gather requirements and understand priorities. We then developed a prototype focused on medication management and delivery and we brought it to an assisted living center to interact with potential older adult users. We interviewed 23 older adults and 5 caregivers who interacted with or observed our prototype to gather feedback. We present quantitative and qualitative data obtained in these interviews, identifying key requirements for robots designed for older adults in the developing world.

Faces are critical in establishing the agency of social robots; however, building expressive mechanical faces is costly and difficult. Instead, many robots built in recent years have faces that are rendered onto a screen. This gives great flexibility in what a robot's face can be and opens up a new design space with which to establish a robot's character and perceived properties. Despite the prevalence of robots with rendered faces, there are no systematic explorations of this design space. Our work aims to fill that gap. We conducted a survey and identified 157 robots with rendered faces and coded them in terms of 76 properties. We present statistics, common patterns, and observations about this data set of faces. Next, we conducted two surveys to understand people's perceptions of rendered robot faces and identify the impact of different face features. Survey results indicate preferences for varying levels of realism and detail in robot faces based on context, and indicate how the presence or absence of specific features affects perception of the face and the types of jobs the face would be appropriate for.

We address the problem of enabling a manipulator to move arbitrary amounts and configurations of dirt on a surface to a goal region using a cleaning tool. We represent this problem as heuristic search with a set of primitive dirt-oriented tool actions. We present dirt and action representations that allow efficient learning and prediction of future dirt states, given the current dirt state and applied action. We also present a method for sampling promising actions based on a clustering of dirt states and heuristics for planning. We demonstrate the effectiveness of our approach on challenging cleaning tasks through implementations on PR2 and Fetch robots.

Arrangement of items on shelves in stores or warehouses is a tedious, repetitive task that can be feasible for robots to perform. The diversity of products that are available in stores and the different setups and preferences of each store makes pre-programming a robot for this task extremely challenging. Instead, our work argues for enabling end-users to customize the robot to their specific objects and setup at deployment time by programming it themselves. To that end, this paper contributes (i) a task representation for shelf arrangements based on a large dataset of grocery store shelf images, (ii) a method for inferring goal configurations from user inputs including demonstrations and direct parameter specifications, and (iii) a system implementation of the proposed approach that allows simultaneously learning task goals and actions. We evaluate our goal inference approach with ten different teaching strategies that combine alternative user inputs in different ways on the large dataset of grocery configurations, as well as with real human teachers through an online user study (N=32). We evaluate our full system implemented on a Fetch mobile manipulator on eight benchmark tasks that demonstrate end-to-end programming and execution of shelf arrangement tasks.

This paper presents four exploratory studies of the potential use of robots for gathering customer feedback in the hospitality industry. To account for the viewpoints of both hotels and guests, we administered need finding interviews at five hotels and an online survey concerning hotel guest experiences with 60 participants. We then conducted the two deployment studies based on deploying software prototypes for Savioke Relay robots we designed to collect customer feedback: (i) a hotel deployment study (three hotels over three months) to explore the feasibility of robot use for gathering customer feedback as well as issues such deployment might pose and (ii) a hotel kitchen deployment study (at Savioke headquarters over three weeks) to explore the role of different robot behaviors (mobility and social attributes) in gathering feedback and understand the customers' thought process in the context that they experience a service. We found that hotels want to collect customer feedback in real-time to disseminate positive feedback immediately and to respond to unhappy customers while they are still on-site. Guests want to inform the hotel staff about their experiences without compromising their convenience and privacy. We also found that the robot users, e.g. hotel staff, use their domain knowledge to increase the response rate to customer feedback surveys at the hotels. Finally, environmental factors, such as robot's location in the building influenced customer response rates more than altering the behaviors of the robot collecting the feedback.

Artificial Intelligence (AI) has had a transformational impact on Human-Robot Interaction (HRI) research over the past decade, enabling work in HRI to develop and investigate robots that can operate autonomously in far more challenging environments and far more complex scenarios than was possible ever before.Beyond laboratory studies, robots that explicitly interact with people as part of their functionality are increasingly being developed, productized, and deployed throughout the world, enabling ecologically valid ethnographic studies of interactions between humans and robots.These advances have been fueled by enabling technologies across many subfields of AI including machine learning, computer vision, task and motion planning, natural language understanding, and dialogue systems.It is not, however, the case that AI research produced polished, ready-off-the-shelf tools that researchers could pick up and effortlessly use to build their envisioned autonomous robot.Rather, the shift has been due to a new, hybrid approach to human-centered robotics research, facilitated by HRI researchers who acquired deep technical skill sets and an influx of AI researchers applying their expertise to HRI problems.More interdiscplinary research teams consisting of formerly AI and HRI researchers also formed, resulting in a vibrant sub-community at the intersection of AI and HRI who came together at the AAAI Fall Symposium on AI for Human-Robot Interaction for the last 4 years.This special issue was encouraged by the continued success and overwhelming popularity of this symposium.Our goal is to exemplify this community's mature, high-quality, and original work, establishing T-HRI as a premier venue for work at the intersection of AI and HRI.Research at this intersection is particularly challenging due to the very need for interdiscplinary, multi-faceted skill sets.AI-HRI researchers need to both innovate in computational techniques and

Hello Barbie, CogniToys Dino, and Amazon Echo are part of a new wave of connected toys and gadgets for the home that listen. Unlike the smartphone, these devices are always on, blending into the background until needed. We conducted interviews with parent-child pairs in which they interacted with Hello Barbie and CogniToys Dino, shedding light on children's expectations of the toys' "intelligence'" and parents' privacy concerns and expectations for parental controls. We find that children were often unaware that others might be able to hear what was said to the toy, and that some parents draw connections between the toys and similar tools not intended as toys (e.g., Siri, Alexa) with which their children already interact. Our findings illuminate people's mental models and experiences with these emerging technologies and will help inform the future designs of interactive, connected toys and gadgets. We conclude with recommendations for parents, designers, and policy makers.

This paper introduces Code3, a system for user-friendly, rapid programming of mobile manipulator robots. The system is designed to let non-roboticists and roboticists alike program end-to-end manipulation tasks. To accomplish this, Code3 provides three integrated components for perception, manipulation, and high-level programming. The perception component helps users define a library of object and scene parts that the robot can later detect. The manipulation component lets users define actions for manipulating objects or scene parts through programming by demonstration. Finally, the high-level programming component provides a drag-and-drop interface with which users can program the logic and control flow to accomplish a task using their previously specified perception and manipulation capabilities. We present findings from an observational user study with non-roboticist programmers (N=10) that demonstrate their ability to quickly learn Code3 and program a PR2 robot to do manipulation tasks. We also demonstrate how the system is expressive enough for an expert to rapidly program highly complex manipulation tasks like playing tic-tac-toe and reconfiguring an object to be graspable.

This paper introduces situated tangible robot programming, whereby a robot is programmed by placing specially designed tangible "blocks" in its workspace. These blocks are used for annotating objects, locations, or regions, and specifying actions and their ordering. The robot compiles a program by detecting blocks and objects in its workspace and grouping them into instructions by solving constraints. We present a proof-of-concept implementation using blocks with unique visual markers in a pick-and-place task domain. Three user studies evaluate the intuitiveness and learnability of situated tangible programming and iterate the block design. We characterize common challenges and gather feedback on how to further improve the design of blocks. Our studies demonstrate that people can interpret, generalize, and create many different situated tangible programs with minimal instruction or with no instruction at all.

While there has been tremendous progress in autonomous robot manipulation, environments with clutter and unknown objects remain challenging particularly for the perception algorithms that support manipulation. This paper adopts a human-aided perception paradigm and investigates alternative interactive segmentation methods to allow users to segment a target object or object part. Through a first user study (N=24) we compare four interactive segmentation methods and characterize the tradeoff between efficiency and accuracy. Next we develop a hybrid segmentation interface and integrate it into an end-to-end human-in-the-loop manipulation system. In a second user study (N=12) we compare the performance of this system to a direct gripper-control system that allows similar manipulation tasks to be performed in challenging scenes. We find that this system enables more efficient manipulation with a lower mental load on the user, while offering a similar task success rate.

Programming robots to do manipulation tasks requires users to specify relevant perceptual landmarks, which include objects, parts of objects, or parts of the workspace. While many techniques have been developed for object detection, few are designed to detect arbitrary parts of objects or of the workspace. This paper presents CustomLandmarks, a flexible tool that lets non-roboticists build their own perceptual detectors for many kinds of landmarks. The system components include a simple 3D interface for specifying landmarks, a novel representation for landmarks, and an algorithm for locating landmarks in new scenes. We evaluate the system's detection performance through systematic experiments and by using the system to aid a PR2 robot with several manipulation tasks. Finally, we present a user study showing that novices to the system are able to understand and use CustomLandmarks quickly, creatively, and effectively.

Programming by Demonstration (PbD) is a promising technique for programming mobile manipulators to perform complex tasks, such as stocking shelves in retail environments. However, programming such tasks purely by demonstration can be cumbersome and time-consuming as they involve many steps and they are different for each item being manipulated. We propose a system that allows programming new tasks with a combination of demonstration and adaptation. This approach eliminates the need to demonstrate repetitions within one task or variations of a task for different items, replacing those demonstrations with a much more time-efficient adaptation procedure. We develop a Graphical User Interface (GUI) that enables the adaptation procedure. This GUI allows grouping, duplicating, removing, reordering, and repositioning parts of a demonstration to adapt and extend it. We implement our approach on a single-armed mobile manipulator. We evaluate our system on several test scenarios with one expert user and four novice users. We demonstrate that the combination of demonstration and adaptation requires substantially less time to program than purely by demonstration.

When surveyed, potential users often report cleaning as a desired robot capability. Cleaning tasks, such as dusting, wiping, or scrubbing, involve applying a tool on a surface. A general-purpose robotic solution to household cleaning needs to address manipulation of the numerous cleaning tools made for different purposes. Finding a universal solution to this manipulation problem is extremely challenging and it is not feasible for developers to pre-program the robot to use every possible tool. Instead, our work seeks to allow end users to program robots by demonstration using their own specific tools. We propose a method to extract a compact representation of a cleaning action from a single demonstration, such that the tool can be applied on different surfaces. The method exploits key insights about tool directionality and constraints placed on the provided demonstration. We demonstrate that our method is able to reliably learn cleaning actions for six different tools and apply those actions on different testing surfaces, even ones smaller than the training surface. Our method reproduces the cleaning performance of the demonstrated trajectory when applied on the training surface and it captures different user preferences.

Robots are becoming popular in Computer Science outreach to K-12 students. Easy-to-program toy robots already exist as commercial educational products. These toys take advantage of the increased interest and engagement resulting from the ability to write code that makes a robot physically move. However, toy robots do not demonstrate the potential of robots to carry out useful everyday tasks. On the other hand, functional robots are often difficult to program even for professional software developers or roboticists. In this work, we apply end-user programming tools for functional robots to the Computer Science outreach context. This experience report describes two offerings of a week-long introductory workshop in which students with various disabilities learned to program a Clearpath Turtlebot, capable of delivering items, interacting with people via touchscreen, and autonomously navigating its environment. We found that the robot and the end-user programming tool that we developed in previous work were successful in provoking interest in Computer Science among both groups of students and in establishing confidence among students that programming is both accessible and interesting. We present key observations from the workshops, lessons learned, and suggestions for readers interested in employing a similar approach.

Building service robots - robots that perform various services in buildings - are becoming more common in large buildings such as hotels and stores. We aim to leverage such robots to serve as guides for blind people. In this paper, we sought to design specifications that detail how a building service robot could interact with and guide a blind person through a building in an effective and socially acceptable way. We conducted participatory design sessions with three designers and five non-designers. Two of the designers and all of the non-designers had a vision disability. Primary features of the design include allowing the user to (1) summon the robot after entering the building, (2) choose from three modes of assistance (Sighted Guide, Escort, and Information Kiosk), and (3) receive information about the building's layout from the robot. We conclude with a discussion of themes and a reflection about our design process that can benefit robot design for blind people in general.

Collaborative robots are quickly gaining momentum in real-world settings. This has motivated many new research questions in human-robot collaboration. In this paper, we address the questions of whether and when a robot should take initiative during joint human-robot task execution. We develop a system capable of autonomously tracking and performing table-top object manipulation tasks with humans and we implement three different initiative models to trigger robot actions. Human-initiated help gives control of robot action timing to the user; robot-initiated reactive help triggers robot assistance when it detects that the user needs help; and robot-initiated proactive help makes the robot help whenever it can. We performed a user study (N=18) to compare these trigger mechanisms in terms of task performance, usage characteristics, and subjective preference. We found that people collaborate best with a proactive robot, yielding better team fluency and high subjective ratings. However, they prefer having control of when the robot should help, rather than working with a reactive robot that only helps when it is needed.

This paper introduces CustomPrograms, a rapid programming system for mobile service robots. With CustomPrograms, roboticists can quickly create new behaviors and try unexplored use cases for commercialization. In our system, the robot has a set of primitive capabilities, such as navigating to a location or interacting with users on a touch screen. Users can then compose these primitives with general-purpose programming language constructs like variables, loops, conditionals, and functions. The programming language is wrapped in a graphical interface. This allows inexperienced or novice programmers to benefit from the system as well. We describe the design and implementation of CustomPrograms on a Savioke Relay robot in detail. Based on interviews conducted with Savioke roboticists, designers, and business people, we learned of several potential new use cases for the robot. We characterize our system's ability to fulfill these use cases. Additionally, we conducted a user study of the interface with Savioke employees and outside programmers. We found that experienced programmers could learn to use the interface and create 3 real-world programs during the 90 minute study. Inexperienced programmers were less likely to create complex programs correctly. We provide an analysis of the errors made during the study, and highlight the most common pieces of feedback we received. Two case studies show how the system was used internally at Savioke and at a major trade show.

Grasping objects in confined environments, such as shelves, fridges, or drawers, is challenging due to the difficulty of avoiding gripper and arm collisions with the surfaces surrounding the object. In this paper we explore the use of a tool to reconfigure objects in such environments so as to make them graspable. The proposed tool has a simple form that allows it to be used in confined environments and a high friction tool tip that enables not only pushing objects but also pulling them. Our approach involves learning predictive models of pre-defined object-directed tool actions from experience. For each action, we train a multi-modal regressor that maps the initial state of an object to changes in that state, such that future states of the object can be estimated. These allow the robot to choose a sequence of tool actions that yield graspable configurations. We demonstrate that our approach enables a PR2 robot to grasp five different objects from different, initially ungraspable, configurations on a shelf.

Autonomous mobile robots will soon become ubiquitous in human-populated environments. Besides their typical applications in fetching, delivery, or escorting, such robots present the opportunity to assist human users in their daily tasks by gathering and reporting up-to-date knowledge about the environment. In this paper, we explore this use case and present an end-to-end framework that enables a mobile robot to answer natural language questions about the state of a large-scale, dynamic environment asked by the inhabitants of that environment. The system parses the question and estimates an initial viewpoint that is likely to contain information for answering the question based on prior environment knowledge. Then, it autonomously navigates towards the viewpoint while dynamically adapting to changes and new information. The output of the system is an image of the most relevant part of the environment that allows the user to obtain an answer to their question. We additionally demonstrate the benefits of a continuously operating information gathering robot by showing how the system can answer retrospective questions about the past state of the world using incidentally recorded sensory data. We evaluate our approach with a custom mobile robot deployed in a university building, with questions collected from occupants of the building. We demonstrate our system's ability to respond to these questions in different environmental conditions.

Though teleoperated robots have become common for more extreme tasks such as bomb diffusion, search-and-rescue, and space exploration, they are not commonly used in human-populated environments for more ordinary tasks such as house cleaning or cooking. This presents near-term opportunities for teleoperated robots in the home. However, a teleoperator's remote presence in a consumer's home presents serious security and privacy risks, and the concerns of end-users about these risks may hinder the adoption of such in-home robots. In this paper, we define and explore the privacy-utility tradeoff for remotely teleoperated robots: as we reduce the quantity or fidelity of visual information received by the teleoperator to preserve the end-user's privacy, we must balance this against the teleoperator's need for sufficient information to successfully carry out tasks. We explore this tradeoff with two surveys that provide a framework for understanding the privacy attitudes of end-users, and with a user study that empirically examines the effect of different filters of visual information on the ability of a teleoperator to carry out a task. Our findings include that respondents do desire privacy protective measures from teleoperators, that respondents prefer certain visual filters from a privacy perspective, and that, for the studied task, we can identify a filter that balances privacy with utility. We make recommendations for in-home teleoperation based on these findings.

Trigger-action programming is a simple programming model that enables users to create rules that automate behavior of smart homes, devices, and online services. Existing trigger-action programming systems, such as if-this-then-that (IFTTT), already have millions of users worldwide; however, their oversimplification limits the expressivity of the programs that can be created. While extensions of IFTTT to allow more complex programs have been proposed, previous work neglects a key distinction between different trigger types (states and events) and action types (instantaneous, extended, and sustained actions). In this paper, we systematically study the impact of these differences through two user studies that reveal: (i) inconsistencies in interpreting the behavior of trigger-action programs and (ii) errors made in creating programs with a desired behavior. Based on a characterization of these issues, we offer recommendations for improving the IFTTT interface so as to mitigate issues that arise from mental model inaccuracies.

General-purpose robots can perform a range of useful tasks in human environments; however, programming them to robustly function in all possible environments that they might encounter is unfeasible. Instead, our research aims to develop robots that can be programmed by its end-users in their context of use, so that the robot needs to robustly function in only one particular environment. This requires intuitive ways in which end-users can program their robot. To that end, this paper contributes a flow-based visual programming language, called RoboFlow, that allows programming of generalizable mobile manipulation tasks. RoboFlow is designed to (i) ensure a robust low-level implementation of program procedures on a mobile manipulator, and (ii) restrict the high-level programming as much as possible to avoid user errors while enabling expressive programs that involve branching, looping, and nesting. We present an implementation of RoboFlow on a PR2 mobile manipulator and demonstrate the generalizability and error handling properties of RoboFlow programs on everyday mobile manipulation tasks in human environments.

Robot Programming by Demonstration (PbD) allows users to program a robot by demonstrating the desired behavior. Providing these demonstrations typically involves moving the robot through a sequence of states, often by physically manipulating it. This requires users to be co-located with the robot and have the physical ability to manipulate it. In this paper, we present a natural language based interface for PbD that removes these requirements and enables hands-free programming. We focus on programming object manipulation actions-our key insight is that such actions can be decomposed into known types of manipulator movements that are naturally described using spatial language; e.g., object reference expressions and prepositions. Our method takes a natural language command and the current world state to infer the intended movement command and its parametrization. We implement this method on a two-armed mobile manipulator and demonstrate the different types of manipulation actions that can be programmed with it. We compare it to a kinesthetic PbD interface and we demonstrate our method's ability to deal with incomplete language.

Advances in mobile robotics have enabled robots that can autonomously operate in human-populated environments. Although primary tasks for such robots might be fetching, delivery, or escorting, they present an untapped potential as information gathering agents that can answer questions for the community of co-inhabitants. In this paper, we seek to better understand requirements for such information gathering robots (InfoBots) from the perspective of the user requesting the information. We present findings from two studies: (i) a user survey conducted in two office buildings and (ii) a 4-day long deployment in one of the buildings, during which inhabitants of the building could ask questions to an InfoBot through a web-based interface. These studies allow us to characterize the types of information that InfoBots can provide for their users.

Humans learn categories of complex objects quickly and from a few examples. Random projection has been suggested as a means to learn and categorize efficiently. We investigate how random projection affects categorization by humans and by very simple neural networks on the same stimuli and categorization tasks, and how this relates to the robustness of categories. We find that (1) drastic reduction in stimulus complexity via random projection does not degrade performance in categorization tasks by either humans or simple neural networks, (2) human accuracy and neural network accuracy are remarkably correlated, even at the level of individual stimuli, and (3) the performance of both is strongly indicated by a natural notion of category robustness.

Programming by Demonstration (PbD) can allow end-users to teach robots new actions simply by demonstrating them. However, learning generalizable actions requires a large number of demonstrations that is unreasonable to expect from end-users. In this paper, we explore the idea of using crowdsourcing to collect action demonstrations from the crowd. We propose a PbD framework in which the end-user provides an initial seed demonstration, and then the robot searches for scenarios in which the action will not work and requests the crowd to fix the action for these scenarios. We use instance-based learning with a simple yet powerful action representation that allows an intuitive visualization of the action. Crowd workers directly interact with these visualizations to fix them. We demonstrate the utility of our approach with a user study involving local crowd workers (N=31) and analyze the collected data and the impact of alternative design parameters so as to inform a real-world deployment of our system.

Allowing end-users to harness the full capability of general purpose robots, requires giving them powerful tools. As the functionality of these tools increase, learning how to use them becomes more challenging. In this paper we investigate the use of instructional materials to support the learnability of a Programming by Demonstration tool. We develop a system that allows users to program complex manipulation skills on a two-armed robot through a spoken dialog interface and by physically moving the robot's arms. We present a user study (N=30) in which participants are left alone with the robot and a user manual, without any prior instructions on how to program the robot. Instead, they are asked to figure it out on their own. We investigate the effect of providing users with an additional written tutorial or an instructional video. We find that videos are most effective in training the user; however, this effect might be superficial and ultimately trial-and-error plays an important role in learning to program the robot. We also find that tutorials can be problematic when the interaction has uncertainty due to speech recognition errors. Overall, the user study demonstrates the effectiveness and learnability of the our system, while providing useful feedback about the dialog design.

Timing plays a role in a range of human-robot interaction scenarios, as humans are highly sensitive to timing and interaction fluency. It is central to spoken dialogue, with turn-taking, interruptions, and hesitation influencing both task efficiency and user affect. Timing is also an important factor in the interpretation and generation of gestures, gaze, facial expressions, and other nonverbal behavior. Beyond communication, temporal synchronization is functionally necessary for sharing resources and physical space, as well as coordinating multi-agent actions. Timing is thus crucial to the success of a broad spectrum of HRI applications, including but not limited to situated dialogue; collaborative manipulation; performance, musical, and entertainment robots; and expressive robot companions. Recent years have seen a growing interest in the HRI community in the various research topics related to human-robot timing. The purpose of this workshop is to explore and discuss theories, computational models, systems, empirical studies, and interdisciplinary insights related to the notion of timing, fluency, and rhythm in human-robot interaction.

Although imitation learning is a powerful technique for robot learning and knowledge acquisition from näıve human users, it often suffers from the need for expensive human demonstrations. In some cases the robot has an insufficient number of useful demonstrations, while in others its learning ability is limited by the number of users it directly interacts with. We propose an approach that overcomes these shortcomings by using crowdsourcing to collect a wider variety of examples from a large pool of human demonstrators online. We present a new goal-based imitation learning framework which utilizes crowdsourcing as a major source of human demonstration data. We demonstrate the effectiveness of our approach experimentally on a scenario where the robot learns to build 2D object models on a table from basic building blocks using knowledge gained from locals and online crowd workers. In addition, we show how the robot can use this knowledge to support human-robot collaboration tasks such as goal inference through object-part classification and missing-part prediction. We report results from a user study involving fourteen local demonstrators and hundreds of crowd workers on 16 different model building tasks.

Robots that can reliably manipulate human tools can do a diverse range of useful tasks in human environments. However, these tools are often difficult to manipulate, particularly given force requirements for applying the tool. This is often due to the mismatch between the robot's gripper and the tool handle designed for human hands. In this paper, we present the design of a low-cost universal tool attachment that makes the tool gripper-friendly. We demonstrate the performance gain provided by the attachment on 10 different tools in the three stages of tool use: grasping the tool, applying the tool, and placing the tool. Our experiments demonstrate that the attachment performs significantly better in all three stages of tool use.

A handover is a complex collaboration, where actors coordinate in time and space to transfer control of an object. This coordination comprises two processes: the physical process of moving to get close enough to transfer the object, and the cognitive process of exchanging information to guide the transfer. Despite this complexity, we humans are capable of performing handovers seamlessly in a wide variety of situations, even when unexpected. This suggests a common procedure that guides all handover interactions. Our goal is to codify that procedure.

A helpful teacher can significantly improve the learning rate of a learning agent. Teaching algorithms have been formally studied within the field of Algorithmic Teaching. These give important insights into how a teacher can select the most informative examples while teachinga new concept. However the field has so far focused purely on classification tasks. In this paper we introducea novel method for optimally teaching sequential decision tasks. We present an algorithm that automatically selects the set of most informative demonstrations andevaluate it on several navigation tasks. Next, we explore the idea of using this algorithm to produce instructions for humans on how to choose examples when teaching sequential decision tasks. We present a user study that demonstrates the utility of such instructions.

Kinesthetic teaching is an approach to providing demonstrations to a robot in Learning from Demonstration whereby a human physically guides a robot to perform a skill. In the common usage of kinesthetic teaching, the robot's trajectory during a demonstration is recorded from start to end. In this paper we consider an alternative, keyframe demonstrations, in which the human provides a sparse set of consecutive keyframes that can be connected to perform the skill. We present a user-study (n=34) comparing the two approaches and highlighting their complementary nature. The study also tests and shows the potential benefits of iterative and adaptive versions of keyframe demonstrations. Finally, we introduce a hybrid method that combines trajectories and keyframes in a single demonstration.

Programming new skills on a robot should take minimal time and effort. One approach to achieve this goal is to allow the robot to ask questions. This idea, called Active Learning, has recently caught a lot of attention in the robotics community. However, it has not been explored from a human-robot interaction perspective. In this paper, we identify three types of questions (label, demonstration and feature queries) and discuss how a robot can use these while learning new skills. Then, we present an experiment on human question asking which characterizes the extent to which humans use these question types. Finally, we evaluate the three question types within a human-robot teaching interaction. We investigate the ease with which different types of questions are answered and whether or not there is a general preference of one type of question over another. Based on our findings from both experiments we provide guidelines for designing question asking behaviors on a robot learner.

We present the hardware design, software architecture, and core algorithms of Herb 2.0, a bimanual mobile manipulator developed at the Personal Robotics Lab at Carnegie Mellon University, Pittsburgh, PA. We have developed Herb 2.0 to perform useful tasks for and with people in human environments. We exploit two key paradigms in human environments: that they have structure that a robot can learn, adapt and exploit, and that they demand general-purpose capability in robotic systems. In this paper, we reveal some of the structure present in everyday environments that we have been able to harness for manipulation and interaction, comment on the particular challenges on working in human spaces, and describe some of the lessons we learned from extensively testing our integrated platform in kitchen and office environments.

For robots to get integrated in daily tasks assisting humans, robot-human interactions will need to reach a level of fluency close to that of human-human interactions. In this paper we address the fluency of robot-human hand-overs. From an observational study with our robot HERB, we identify the key problems with a baseline hand-over action. We find that the failure to convey the intention of handing over causes delays in the transfer, while the lack of an intuitive signal to indicate timing of the hand-over causes early, unsuccessful attempts to take the object. We propose to address these problems with the use of spatial contrast, in the form of distinct hand-over poses, and temporal contrast, in the form of unambiguous transitions to the hand-over pose. We conduct a survey to identify distinct hand-over poses, and determine variables of the pose that have most communicative potential for the intent of handing over. We present an experiment that analyzes the effect of the two types of contrast on the fluency of hand-overs. We find that temporal contrast is particularly useful in improving fluency by eliminating early attempts of the human.

One goal of assistive robotics is to design interactive robots that can help disabled people with tasks such as fetching objects. When people do this task, they coordinate their movements closely with receivers. We investigated how a robot should fetch and give household objects to a person. To develop a model for the robot, we first studied trained dogs and person-to-person handoffs. Our findings suggest two models of handoff that differ in their predictability and adaptivity.

Handing over objects to humans is an essential capability for assistive robots. While there are infinite ways to hand an object, robots should be able to choose the one that is best for the human. In this paper we focus on choosing the robot and object configuration at which the transfer of the object occurs, i.e. the hand-over configuration. We advocate the incorporation of user preferences in choosing hand-over configurations. We present a user study in which we collect data on human preferences and a human-robot interaction experiment in which we compare hand-over configurations learned from human examples against configurations planned using a kinematic model of the human. We find that the learned configurations are preferred in terms of several criteria, however planned configurations provide better reachability. Additionally, we find that humans prefer hand-overs with default orientations of objects and we identify several latent variables about the robot's arm that capture significant human preferences. These findings point towards planners that can generate not only optimal but also preferable hand-over configurations for novel objects.

This research aims to enable robots to learn from human teachers. Motivated by human social learning, we believe that a transparent learning process can help guide the human teacher to provide the most informative instruction. We believe active learning is an inherently transparent machine learning approach because the learner formulates queries to the oracle that reveal information about areas of uncertainty in the underlying model. In this work, we implement active learning on the Simon robot in the form of nonverbal gestures that query a human teacher about a demonstration within the context of a social dialogue. Our preliminary pilot study data show potential for transparency through active learning to improve the accuracy and efficiency of the teaching process. However, our data also seem to indicate possible undesirable effects from the human teacher's perspective regarding balance of the interaction. These preliminary results argue for control strategies that balance leading and following during a social learning interaction.

A general learning task for a robot in a new environment is to learn about objects and what actions/effects they afford. To approach this, we look at ways that a human partner can intuitively help the robot learn, Socially Guided Machine Learning. We present experiments conducted with our robot, Junior, and make six observations characterizing how people approached teaching about objects. We show that Junior successfully used transparency to mitigate errors. Finally, we present the impact of "social" versus "non-social" data sets when training SVM classifiers.

Social learning in robotics has largely focused on imitation learning. Here we take a broader view and are interested in the multifaceted ways that a social partner can influence the learning process. We implement four social learning mechanisms on a robot: stimulus enhancement, emulation, mimicking, and imitation, and illustrate the computational benefits of each. In particular, we illustrate that some strategies are about directing the attention of the learner to objects and others are about actions. Taken together these strategies form a rich repertoire allowing social learners to use a social partner to greatly impact their learning process. We demonstrate these results in simulation and with physical robot `playmates'.

"Developmental robotics" proposes that, instead of trying to build a robot that shows intelligence once and for all, what one must do is to build robots that can develop. These robots should be equipped with behaviors that are simple but enough to bootstrap the system. Then, as the robot interacts with its environment, it should display increasingly complex behaviors. In this paper, we propose such a development scheme for a mobile robot. J.J. Gibson's concept of "affordances" and a formalization of this concept provides the basis of this development scheme. We show that an autonomous robot can start with pre-coded primitive behaviors, and as it executes its behaviors randomly in an environment, it can learn the affordance relations between the environment and its behaviors. We then present two ways of using these learned structures, in achieving more complex, intentional behaviors. In the first case, the sequencing of these primitive behaviors are such that new more complex behaviors emerge. In the second case, the robot makes a "blending" of its pre-coded primitive behaviors to create new behaviors.

We are interested in how the concept of affordances can affect our view to autonomous robot control, and how the results obtained from autonomous robotics can be reflected back upon the discussion and studies on the concept of affordances. In this paper, we studied how a mobile robot, equipped with a 3D laser scanner, can learn to perceive the traversability affordance and use it to wander in a room tilled with spheres, cylinders and boxes. The results showed that after learning, the robot can wander around avoiding contact with non-traversable objects (i.e. boxes, upright cylinders, or lying cylinders in certain orientation), but moving over traversable objects (such as spheres, and lying cylinders in a rollable orientation with respect to the robot) rolling them out of its way. We have shown that for each action approximately 1% of the perceptual features were relevant to determine whether it is afforded or not and that these relevant features are positioned in certain regions of the range image. The experiments are conducted both using a physics-based simulator and on a real robot.

In this paper, we studied how a mobile robot equipped with a 3D laser scanner can start from primitive behaviors and learn to use them to achieve goal-directed behaviors. For this purpose, we propose a learning scheme that is based on the concept of "affordances", where the robot first learns about the different kind of effects it can create in the environment and then links these effects with the perception of the initial environment and the executed primitive behavior. It uses these learned relations to create certain effects in the environment and achieve more complex behaviors.