Programming by Demonstration (PbD) lets users with little technical background program a wide variety of manipulation tasks for robots, but it should be as intuitive as possible for users while requiring as little time as possible. In this paper, we present a Programming by Demonstration system that synthesizes manipulation programs from a single observed demonstration, allowing users to program new tasks for a robot simply by performing the task once themselves. A human-in-the-loop interface helps users make corrections to the perceptual state as needed. We introduce Object Interaction Programs as a representation of multi-object, bimanual manipulation tasks and present algorithms for extracting programs from observed demonstrations and transferring programs to a robot to perform the task in a new scene. We demonstrate the expressivity and generalizability of our approach through an evaluation on a benchmark of complex tasks.

This paper introduces Code3, a system for user-friendly, rapid programming of mobile manipulator robots. The system is designed to let non-roboticists and roboticists alike program end-to-end manipulation tasks. To accomplish this, Code3 provides three integrated components for perception, manipulation, and high-level programming. The perception component helps users define a library of object and scene parts that the robot can later detect. The manipulation component lets users define actions for manipulating objects or scene parts through programming by demonstration. Finally, the high-level programming component provides a drag-and-drop interface with which users can program the logic and control flow to accomplish a task using their previously specified perception and manipulation capabilities. We present findings from an observational user study with non-roboticist programmers (N=10) that demonstrate their ability to quickly learn Code3 and program a PR2 robot to do manipulation tasks. We also demonstrate how the system is expressive enough for an expert to rapidly program highly complex manipulation tasks like playing tic-tac-toe and reconfiguring an object to be graspable.

Programming robots to do manipulation tasks requires users to specify relevant perceptual landmarks, which include objects, parts of objects, or parts of the workspace. While many techniques have been developed for object detection, few are designed to detect arbitrary parts of objects or of the workspace. This paper presents CustomLandmarks, a flexible tool that lets non-roboticists build their own perceptual detectors for many kinds of landmarks. The system components include a simple 3D interface for specifying landmarks, a novel representation for landmarks, and an algorithm for locating landmarks in new scenes. We evaluate the system's detection performance through systematic experiments and by using the system to aid a PR2 robot with several manipulation tasks. Finally, we present a user study showing that novices to the system are able to understand and use CustomLandmarks quickly, creatively, and effectively.

This paper presents findings from two deployments of an autonomous mobile robot in older adult low income Supportive Apartment Living (SAL) facilities. Design guidelines for the robot hardware and software were based on query of clinicians, caregivers and older adults through focus groups, member checks and surveys, to identify what each group believed to be the most important daily activities for older adults to accomplish physically, mentally and socially. After data analysis, hydration and walking encouragement were found to be critical daily activities, becoming the focus of our deployments. The aim of the deployments was to understand the efficacy of human-robot interaction and identify ways to enhance the robot design and programming. Through observation of older adults interacting with the robot and post-interaction surveys filled out by the older adults, conclusions were drawn for further advancement of the robot development to be tested in future deployments. Results overall indicated high perceived usefulness and growing acceptance of the robot by older adults with increased interactions.

Robots are becoming popular in Computer Science outreach to K-12 students. Easy-to-program toy robots already exist as commercial educational products. These toys take advantage of the increased interest and engagement resulting from the ability to write code that makes a robot physically move. However, toy robots do not demonstrate the potential of robots to carry out useful everyday tasks. On the other hand, functional robots are often difficult to program even for professional software developers or roboticists. In this work, we apply end-user programming tools for functional robots to the Computer Science outreach context. This experience report describes two offerings of a week-long introductory workshop in which students with various disabilities learned to program a Clearpath Turtlebot, capable of delivering items, interacting with people via touchscreen, and autonomously navigating its environment. We found that the robot and the end-user programming tool that we developed in previous work were successful in provoking interest in Computer Science among both groups of students and in establishing confidence among students that programming is both accessible and interesting. We present key observations from the workshops, lessons learned, and suggestions for readers interested in employing a similar approach.

This paper introduces CustomPrograms, a rapid programming system for mobile service robots. With CustomPrograms, roboticists can quickly create new behaviors and try unexplored use cases for commercialization. In our system, the robot has a set of primitive capabilities, such as navigating to a location or interacting with users on a touch screen. Users can then compose these primitives with general-purpose programming language constructs like variables, loops, conditionals, and functions. The programming language is wrapped in a graphical interface. This allows inexperienced or novice programmers to benefit from the system as well. We describe the design and implementation of CustomPrograms on a Savioke Relay robot in detail. Based on interviews conducted with Savioke roboticists, designers, and business people, we learned of several potential new use cases for the robot. We characterize our system's ability to fulfill these use cases. Additionally, we conducted a user study of the interface with Savioke employees and outside programmers. We found that experienced programmers could learn to use the interface and create 3 real-world programs during the 90 minute study. Inexperienced programmers were less likely to create complex programs correctly. We provide an analysis of the errors made during the study, and highlight the most common pieces of feedback we received. Two case studies show how the system was used internally at Savioke and at a major trade show.

Though teleoperated robots have become common for more extreme tasks such as bomb diffusion, search-and-rescue, and space exploration, they are not commonly used in human-populated environments for more ordinary tasks such as house cleaning or cooking. This presents near-term opportunities for teleoperated robots in the home. However, a teleoperator's remote presence in a consumer's home presents serious security and privacy risks, and the concerns of end-users about these risks may hinder the adoption of such in-home robots. In this paper, we define and explore the privacy-utility tradeoff for remotely teleoperated robots: as we reduce the quantity or fidelity of visual information received by the teleoperator to preserve the end-user's privacy, we must balance this against the teleoperator's need for sufficient information to successfully carry out tasks. We explore this tradeoff with two surveys that provide a framework for understanding the privacy attitudes of end-users, and with a user study that empirically examines the effect of different filters of visual information on the ability of a teleoperator to carry out a task. Our findings include that respondents do desire privacy protective measures from teleoperators, that respondents prefer certain visual filters from a privacy perspective, and that, for the studied task, we can identify a filter that balances privacy with utility. We make recommendations for in-home teleoperation based on these findings.

Trigger-action programming is a simple programming model that enables users to create rules that automate behavior of smart homes, devices, and online services. Existing trigger-action programming systems, such as if-this-then-that (IFTTT), already have millions of users worldwide; however, their oversimplification limits the expressivity of the programs that can be created. While extensions of IFTTT to allow more complex programs have been proposed, previous work neglects a key distinction between different trigger types (states and events) and action types (instantaneous, extended, and sustained actions). In this paper, we systematically study the impact of these differences through two user studies that reveal: (i) inconsistencies in interpreting the behavior of trigger-action programs and (ii) errors made in creating programs with a desired behavior. Based on a characterization of these issues, we offer recommendations for improving the IFTTT interface so as to mitigate issues that arise from mental model inaccuracies.

A clear contradiction exists between cytotoxic in-vitro studies demonstrating effectiveness of Gemcitabine to curtail pancreatic cancer and in-vivo studies failing to show Gemcitabine as an effective treatment. The outcome of chemotherapy in metastatic stages, where surgery is no longer viable, shows a 5-year survival <5%. It is apparent that in-vitro experiments, no matter how well designed, may fail to adequately represent the complex in-vivo microenvironmental and phenotypic characteristics of the cancer, including cell proliferation and apoptosis. We evaluate in-vitro cytotoxic data as an indicator of in-vivo treatment success using a mathematical model of tumor growth based on a dimensionless formulation describing tumor biology. Inputs to the model are obtained under optimal drug exposure conditions in-vitro. The model incorporates heterogeneous cell proliferation and death caused by spatial diffusion gradients of oxygen/nutrients due to inefficient vascularization and abundant stroma, and thus is able to simulate the effect of the microenvironment as a barrier to effective nutrient and drug delivery. Analysis of the mathematical model indicates the pancreatic tumors to be mostly resistant to Gemcitabine treatment in-vivo. The model results are confirmed with experiments in live mice, which indicate uninhibited tumor proliferation and metastasis with Gemcitabine treatment. By extracting mathematical model parameter values for proliferation and death from monolayer in-vitro cytotoxicity experiments with pancreatic cancer cells, and simulating the effects of spatial diffusion, we use the model to predict the drug response in-vivo, beyond what would have been expected from sole consideration of the cancer intrinsic resistance. We conclude that this integrated experimental/computational approach may enhance understanding of pancreatic cancer behavior and its response to various chemotherapies, and, further, that such an approach could predict resistance based on pharmacokinetic measurements with the goal to maximize effective treatment strategies.