We address the problem of enabling a manipulator to move arbitrary amounts and configurations of dirt on a surface to a goal region using a cleaning tool. We represent this problem as heuristic search with a set of primitive dirt-oriented tool actions. We present dirt and action representations that allow efficient learning and prediction of future dirt states, given the current dirt state and applied action. We also present a method for sampling promising actions based on a clustering of dirt states and heuristics for planning. We demonstrate the effectiveness of our approach on challenging cleaning tasks through implementations on PR2 and Fetch robots.

While there has been tremendous progress in autonomous robot manipulation, environments with clutter and unknown objects remain challenging particularly for the perception algorithms that support manipulation. This paper adopts a human-aided perception paradigm and investigates alternative interactive segmentation methods to allow users to segment a target object or object part. Through a first user study (N=24) we compare four interactive segmentation methods and characterize the tradeoff between efficiency and accuracy. Next we develop a hybrid segmentation interface and integrate it into an end-to-end human-in-the-loop manipulation system. In a second user study (N=12) we compare the performance of this system to a direct gripper-control system that allows similar manipulation tasks to be performed in challenging scenes. We find that this system enables more efficient manipulation with a lower mental load on the user, while offering a similar task success rate.

Programming by Demonstration (PbD) is a promising technique for programming mobile manipulators to perform complex tasks, such as stocking shelves in retail environments. However, programming such tasks purely by demonstration can be cumbersome and time-consuming as they involve many steps and they are different for each item being manipulated. We propose a system that allows programming new tasks with a combination of demonstration and adaptation. This approach eliminates the need to demonstrate repetitions within one task or variations of a task for different items, replacing those demonstrations with a much more time-efficient adaptation procedure. We develop a Graphical User Interface (GUI) that enables the adaptation procedure. This GUI allows grouping, duplicating, removing, reordering, and repositioning parts of a demonstration to adapt and extend it. We implement our approach on a single-armed mobile manipulator. We evaluate our system on several test scenarios with one expert user and four novice users. We demonstrate that the combination of demonstration and adaptation requires substantially less time to program than purely by demonstration.

When surveyed, potential users often report cleaning as a desired robot capability. Cleaning tasks, such as dusting, wiping, or scrubbing, involve applying a tool on a surface. A general-purpose robotic solution to household cleaning needs to address manipulation of the numerous cleaning tools made for different purposes. Finding a universal solution to this manipulation problem is extremely challenging and it is not feasible for developers to pre-program the robot to use every possible tool. Instead, our work seeks to allow end users to program robots by demonstration using their own specific tools. We propose a method to extract a compact representation of a cleaning action from a single demonstration, such that the tool can be applied on different surfaces. The method exploits key insights about tool directionality and constraints placed on the provided demonstration. We demonstrate that our method is able to reliably learn cleaning actions for six different tools and apply those actions on different testing surfaces, even ones smaller than the training surface. Our method reproduces the cleaning performance of the demonstrated trajectory when applied on the training surface and it captures different user preferences.

Robots are becoming popular in Computer Science outreach to K-12 students. Easy-to-program toy robots already exist as commercial educational products. These toys take advantage of the increased interest and engagement resulting from the ability to write code that makes a robot physically move. However, toy robots do not demonstrate the potential of robots to carry out useful everyday tasks. On the other hand, functional robots are often difficult to program even for professional software developers or roboticists. In this work, we apply end-user programming tools for functional robots to the Computer Science outreach context. This experience report describes two offerings of a week-long introductory workshop in which students with various disabilities learned to program a Clearpath Turtlebot, capable of delivering items, interacting with people via touchscreen, and autonomously navigating its environment. We found that the robot and the end-user programming tool that we developed in previous work were successful in provoking interest in Computer Science among both groups of students and in establishing confidence among students that programming is both accessible and interesting. We present key observations from the workshops, lessons learned, and suggestions for readers interested in employing a similar approach.

Grasping objects in confined environments, such as shelves, fridges, or drawers, is challenging due to the difficulty of avoiding gripper and arm collisions with the surfaces surrounding the object. In this paper we explore the use of a tool to reconfigure objects in such environments so as to make them graspable. The proposed tool has a simple form that allows it to be used in confined environments and a high friction tool tip that enables not only pushing objects but also pulling them. Our approach involves learning predictive models of pre-defined object-directed tool actions from experience. For each action, we train a multi-modal regressor that maps the initial state of an object to changes in that state, such that future states of the object can be estimated. These allow the robot to choose a sequence of tool actions that yield graspable configurations. We demonstrate that our approach enables a PR2 robot to grasp five different objects from different, initially ungraspable, configurations on a shelf.

We describe our experience exhibiting a human-size robot in a museum, encouraging visitors to interact with the robot and even program it to perform a sequence of timed poses. At the museum, users' programs were run on a real robot for all to see. The installation attracted and engaged visitors from age two to adult. The most intuitive of our interfaces was equally captivating for young and older visitors. We present the pros and cons of our interfaces and the engagement of visitors at the exhibit as lessons for other exhibitors who aim to achieve active prolonged engagement with robots in museum settings.