We present ForceSight, a system for text-guided mobile manipulation that predicts visual-force goals using a text-conditioned vision transformer. Given a single RGBD image and a text prompt, ForceSight determines a target end-effector pose in the camera frame (kinematic goal) and the associated forces (force goal). Together, these two components form a visual-force goal. Prior work has demonstrated that deep models outputting human-interpretable kinematic goals can enable dexterous manipulation by real robots. Forces are critical to manipulation, yet have typically been relegated to low-level execution in these systems. When deployed on a mobile manipulator equipped with an eye-in-hand RGBD camera, ForceSight performed tasks such as precision grasps, drawer opening, and object handovers with an 81% success rate in unseen environments with object instances that differed significantly from the training data. In a separate experiment, relying exclusively on visual servoing and ignoring force goals dropped the success rate from 90% to 45%, demonstrating that force goals can significantly enhance performance. The appendix, videos, code, and trained models are available at https://force-sight.github.io/.

Physical therapy (PT) is a key component of many rehabilitation regimens, such as treatments for Parkinson’s disease (PD). However, there are shortages of physical therapists and adherence to self-guided PT is low. Robots have the potential to support physical therapists and increase adherence to self-guided PT, but prior robotic systems have been large and immobile, which can be a barrier to use in homes and clinics. We present Stretch with Stretch (SWS), a novel robotic system for leading stretching exercise games for older adults with PD. SWS consists of a compact and lightweight mobile manipulator (Hello Robot Stretch RE1) that visually and verbally guides users through PT exercises. The robot’s soft end effector serves as a target that users repetitively reach towards and press with a hand, foot, or knee. For each exercise, target locations are customized for the individual via a visually estimated kinematic model, a haptically estimated range of motion, and the person’s exercise performance. The system includes sound effects and verbal feedback from the robot to keep users engaged throughout a session and augment physical exercise with cognitive exercise. We conducted a user study for which people with PD (n = 10) performed 6 exercises with the system. Participants perceived the SWS to be useful and easy to use. They also reported mild to moderate perceived exertion (RPE).

How easy is it to sneak up on a robot? We examine whether we can detect people using only the incidental sounds they produce as they move, even when they try to be quiet. To do so, we first collect a robotic dataset of high-quality 4-channel audio paired with 360° RGB data of people moving in different indoor settings. Using this dataset, we train models to predict if there is a moving person nearby and then their location using only audio. We implement our method on a robot, allowing it to track a single person moving quietly using only passive audio sensing. For demonstration videos, see our project page.

Touch plays a fundamental role in manipulation for humans; however, machine perception of contact and pressure typically requires invasive sensors. Recent research has shown that deep models can estimate hand pressure based on a single RGB image. However, evaluations have been limited to controlled settings since collecting diverse data with ground-truth pressure measurements is difficult. We present a novel approach that enables diverse data to be captured with only an RGB camera and a cooperative participant. Our key insight is that people can be prompted to apply pressure in a certain way, and this prompt can serve as a weak label to supervise models to perform well under varied conditions. We collect a novel dataset with 51 participants making fingertip contact with diverse objects. Our network, PressureVision++, outperforms human annotators and prior work. We also demonstrate an application of PressureVision++ to mixed reality where pressure estimation allows everyday surfaces to be used as arbitrary touch-sensitive interfaces. Code, data, and models are available online .1

Robotic manipulation can benefit from wrist-mounted force/torque (F/T) sensors, but conventional F/T sensors can be expensive, difficult to install, and damaged by high loads. We present Visual Force/Torque Sensing (VFTS), a method that visually estimates the 6-axis F/T measurement that would be reported by a conventional F/T sensor. In contrast to approaches that sense loads using internal cameras placed behind soft exterior surfaces, our approach uses an external camera with a fisheye lens that observes a soft gripper. VFTS includes a deep learning model that takes a single RGB image as input and outputs a 6-axis F/T estimate. We trained the model with sensor data collected while teleoperating a robot (Stretch RE1 from Hello Robot Inc.) to perform manipulation tasks. VFTS outperformed F/T estimates based on motor currents, generalized to a novel home environment, and supported three autonomous tasks relevant to healthcare: grasping a blanket, pulling a blanket over a manikin, and cleaning a manikin's limbs. VFTS also performed well with a manually operated pneumatic gripper. Overall, our results suggest that an external camera observing a soft gripper can perform useful visual force/torque sensing for a variety of manipulation tasks.

Sensing contact pressure applied by a gripper can benefit autonomous and teleoperated robotic manipulation, but adding tactile sensors to a gripper's surface can be difficult or impractical. If a gripper visibly deforms, contact pressure can be visually estimated using images from an external camera that observes the gripper. While researchers have demonstrated this capability in controlled laboratory settings, prior work has not addressed challenges associated with visual pressure estimation in the wild, where lighting, surfaces, and other factors vary widely. We present a deep learning model and associated methods that enable visual pressure estimation under widely varying conditions. Our model, Visual Pressure Estimation for Robots (ViPER), takes an image from an eye-in-hand camera as input and outputs an image representing the pressure applied by a soft gripper. Our key insight is that force/torque sensing can be used as a weak label to efficiently collect training data in settings where pressure measurements would be difficult to ob-tain. When trained on this weakly labeled data combined with fully labeled data that includes pressure measurements, ViPER outperforms prior methods, enables precision manipulation in cluttered settings, and provides accurate estimates for unseen conditions relevant to in-home use.

Contact pressure between the human body and its surroundings has important implications. For example, it plays a role in comfort, safety, posture, and health. We present a method that infers contact pressure between a human body and a mattress from a depth image. Specifically, we focus on using a depth image from a downward facing camera to infer pressure on a body at rest in bed occluded by bedding, which is directly applicable to the prevention of pressure injuries in healthcare. Our approach involves augmenting a real dataset with synthetic data generated via a soft-body physics simulation of a human body, a mattress, a pressure sensing mat, and a blanket. We introduce a novel deep network that we trained on an augmented dataset and evaluated with real data. The network contains an embedded human body mesh model and uses a white-box model of depth and pressure image generation. Our network successfully infers body pose, outperforming prior work. It also infers contact pressure across a 3D mesh model of the human body, which is a novel capability, and does so in the presence of occlusion from blankets.

Toward the goal of robots performing robust and intelligent physical interactions with people, it is crucial that robots are able to accurately sense the human body, follow trajectories around the body, and track human motion. This study introduces a capacitive servoing control scheme that allows a robot to sense and navigate around human limbs during close physical interactions. Capacitive servoing leverages temporal measurements from a multielectrode capacitive sensor array mounted on a robot's end effector to estimate the relative position and orientation (pose) of a nearby human limb. Capacitive servoing then uses these human pose estimates from a data-driven pose estimator within a feedback control loop in order to maneuver the robot's end effector around the surface of a human limb. We provide a design overview of capacitive sensors for human–robot interaction and then investigate the performance and generalization of capacitive servoing through an experiment with 12 human participants. The results indicate that multidimensional capacitive servoing enables a robot's end effector to move proximally or distally along human limbs while adapting to human pose. Using a cross-validation experiment, results further show that capacitive servoing generalizes well across people with different body size.

Mobile manipulators for indoor human environments can serve as versatile devices that perform a variety of tasks, yet adoption of this technology has been limited. Reducing size, weight, and cost could facilitate adoption, but risks restricting capabilities. We present a novel design that reduces size, weight, and cost, while supporting a variety of tasks. The core design consists of a two-wheeled differential-drive mobile base, a lift, and a telescoping arm configured to achieve Cartesian motion at the end of the arm. Design extensions include a 1 degree-of-freedom (DOF) wrist to stow a tool, a 2-DOF dexterous wrist to pitch and roll a tool, and a compliant gripper. We justify our design with anthropometry and mathematical models of static stability. We also provide empirical support from teleoperating and autonomously controlling a commercial robot based on our design (the Stretch RE1 from Hello Robot Inc.) to perform tasks in real homes.

Soft robotic grippers facilitate contact-rich manipulation, including robust grasping of varied objects. Yet the beneficial compliance of a soft gripper also results in significant deformation that can make precision manipulation challenging. We present visual pressure estimation & control (VPEC), a method that infers pressure applied by a soft gripper using an RGB image from an external camera. We provide results for visual pressure inference when a pneumatic gripper and a tendon-actuated gripper make contact with a flat surface. We also show that VPEC enables precision manipulation via closed-loop control of inferred pressure images. In our evaluation, a mobile manipulator (Stretch RE1 from Hello Robot) uses visual servoing to make contact at a desired pressure; follow a spatial pressure trajectory; and grasp small low-profile objects, including a microSD card, a penny, and a pill. Overall, our results show that visual estimates of applied pressure can enable a soft gripper to perform precision manipulation.

Physical contact between hands and objects plays a critical role in human grasps. We show that optimizing the pose of a hand to achieve expected contact with an object can improve hand poses inferred via image-based methods. Given a hand mesh and an object mesh, a deep model trained on ground truth contact data infers desirable contact across the surfaces of the meshes. Then, ContactOpt efficiently optimizes the pose of the hand to achieve desirable contact using a differentiable contact model. Notably, our contact model encourages mesh interpenetration to approximate deformable soft tissue in the hand. In our evaluations, our methods result in grasps that better match ground truth contact, have lower kinematic error, and are significantly preferred by human participants. Code and models are available online1.

People spend a substantial part of their lives at rest in bed. 3D human pose and shape estimation for this activity would have numerous beneficial applications, yet line-of-sight perception is complicated by occlusion from bedding. Pressure sensing mats are a promising alternative, but training data is challenging to collect at scale. We describe a physics-based method that simulates human bodies at rest in a bed with a pressure sensing mat, and present PressurePose, a synthetic dataset with 206K pressure images with 3D human poses and shapes. We also present PressureNet, a deep learning model that estimates human pose and shape given a pressure image and gender. PressureNet incorporates a pressure map reconstruction (PMR) network that models pressure image generation to promote consistency between estimated 3D body models and pressure image input. In our evaluations, PressureNet performed well with real data from participants in diverse poses, even though it had only been trained with synthetic data. When we ablated the PMR network, performance dropped substantially.

Autonomous robots have the potential to serve as versatile caregivers that improve quality of life for millions of people worldwide. Yet, conducting research in this area presents numerous challenges, including the risks of physical interaction between people and robots. Physics simulations have been used to optimize and train robots for physical assistance, but have typically focused on a single task. In this paper, we present Assistive Gym, an open source physics simulation framework for assistive robots that models multiple tasks. It includes six simulated environments in which a robotic manipulator can attempt to assist a person with activities of daily living (ADLs): itch scratching, drinking, feeding, body manipulation, dressing, and bathing. Assistive Gym models a person's physical capabilities and preferences for assistance, which are used to provide a reward function. We present baseline policies trained using reinforcement learning for four different commercial robots in the six environments. We demonstrate that modeling human motion results in better assistance and we compare the performance of different robots. Overall, we show that Assistive Gym is a promising tool for assistive robotics research.

Material recognition can help inform robots about how to properly interact with and manipulate real-world objects. In this paper, we present a multimodal sensing technique, leveraging near-infrared spectroscopy and close-range high resolution texture imaging, that enables robots to estimate the materials of household objects. We release a dataset of high resolution texture images and spectral measurements collected from a mobile manipulator that interacted with 144 house-hold objects. We then present a neural network architecture that learns a compact multimodal representation of spectral measurements and texture images. When generalizing material classification to new objects, we show that this multimodal representation enables a robot to recognize materials with greater performance as compared to prior state-of-the-art approaches. Finally, we present how a robot can combine this high resolution local sensing with images from the robot's head-mounted camera to achieve accurate material classification over a scene of objects on a table.

Versatile robotic caregivers could benefit millions of people worldwide, including older adults and people with disabilities. Recent work has explored how robotic caregivers can learn to interact with people through physics simulations, yet transferring what has been learned to real robots remains challenging. Virtual reality (VR) has the potential to help bridge the gap between simulations and the real world. We present Assistive VR Gym (AVR Gym), which enables real people to interact with virtual assistive robots. We also provide evidence that AVR Gym can help researchers improve the performance of simulation-trained assistive robots with real people. Prior to AVR Gym, we trained robot control policies (Original Policies) solely in simulation for four robotic caregiving tasks (robot-assisted feeding, drinking, itch scratching, and bed bathing) with two simulated robots (PR2 from Willow Garage and Jaco from Kinova). With AVR Gym, we developed Revised Policies based on insights gained from testing the Original policies with real people. Through a formal study with eight participants in AVR Gym, we found that the Original policies performed poorly, the Revised policies performed significantly better, and that improvements to the biomechanical models used to train the Revised policies resulted in simulated people that better match real participants. Notably, participants significantly dis-agreed that the Original policies were successful at assistance, but significantly agreed that the Revised policies were successful at assistance. Overall, our results suggest that VR can be used to improve the performance of simulation-trained control policies with real people without putting people at risk, thereby serving as a valuable stepping stone to real robotic assistance.

We present a method of training character manipulation of amorphous materials such as those often used in cooking. Common examples of amorphous materials include granular materials (salt, uncooked rice), fluids (honey), and visco-plastic materials (sticky rice, softened butter). A typical task is to spread a given material out across a flat surface using a tool such as a scraper or knife. We use reinforcement learning to train our controllers to manipulate materials in various ways. The training is performed in a physics simulator that uses position-based dynamics of particles to simulate the materials to be manipulated. The neural network control policy is given observations of the material (e.g. a low-resolution density map), and the policy outputs actions such as rotating and translating the knife. We demonstrate policies that have been successfully trained to carry out the following tasks: spreading, gathering, and flipping. We produce a final animation by using inverse kinematics to guide a character's arm and hand to match the motion of the manipulation tool such as a knife or a frying pan.

Grasping and manipulating objects is an important human skill. Since hand-object contact is fundamental to grasping, capturing it can lead to important insights. However, observing contact through external sensors is challenging because of occlusion and the complexity of the human hand. We present ContactDB, a novel dataset of contact maps for household objects that captures the rich hand-object contact that occurs during grasping, enabled by use of a thermal camera. Participants in our study grasped 3D printed objects with a post-grasp functional intent. ContactDB includes 3750 3D meshes of 50 household objects textured with contact maps and 375K frames of synchronized RGB-D+thermal images. To the best of our knowledge, this is the first large-scale dataset that records detailed contact maps for human grasps. Analysis of this data shows the influence of functional intent and object size on grasping, the tendency to touch/avoid `active areas', and the high frequency of palm and proximal finger contact. Finally, we train state-of-the art image translation and 3D convolution algorithms to predict diverse contact patterns from object shape. Data, code and models are available at https://contactdb.cc.gatech.edu.

Robot-assisted dressing offers an opportunity to benefit the lives of many people with disabilities, such as some older adults. However, robots currently lack common sense about the physical implications of their actions on people. The physical implications of dressing are complicated by non-rigid garments, which can result in a robot indirectly applying high forces to a person's body. We present a deep recurrent model that, when given a proposed action by the robot, predicts the forces a garment will apply to a person's body. We also show that a robot can provide better dressing assistance by using this model with model predictive control. The predictions made by our model only use haptic and kinematic observations from the robot's end effector, which are readily attainable. Collecting training data from real world physical human-robot interaction can be time consuming, costly, and put people at risk. Instead, we train our predictive model using data collected in an entirely self-supervised fashion from a physics-based simulation. We evaluated our approach with a PR2 robot that attempted to pull a hospital gown onto the arms of 10 human participants. With a 0.2s prediction horizon, our controller succeeded at high rates and lowered applied force while navigating the garment around a persons fist and elbow without getting caught. Shorter prediction horizons resulted in significantly reduced performance with the sleeve catching on the participants' fists and elbows, demonstrating the value of our model's predictions. These behaviors of mitigating catches emerged from our deep predictive model and the controller objective function, which primarily penalizes high forces.

By briefly heating the local environment with a heat lamp and observing what happens with a thermal camera, robots could potentially infer properties of their surroundings. However, this form of active thermography introduces large signal variations compared to traditional active thermography, which has typically been used to characterize small regions of materials in carefully controlled settings. We demonstrate that a data-driven approach with modern machine learning methods can be used to classify material samples over relatively large surface areas and variable distances. We also introduce the use of z-normalization to improve material classification and reduce variation due to distance and heating intensity. Our best performing algorithm achieved an overall accuracy of 77.7% for multi-class classification among 12 materials placed at varying distances (20 cm, 30 cm, and 40 cm). The observations were made for 5 seconds with 1s of heating and 4s of cooling. We also provide a demonstration of performance with a multi-material scene.

Robots have the potential to assist people in bed, such as in healthcare settings, yet bedding materials like sheets and blankets can make observation of the human body difficult for robots. A pressure-sensing mat on a bed can provide pressure images that are relatively insensitive to bedding materials. However, prior work on estimating human pose from pressure images has been restricted to 2D pose estimates and flat beds. In this work, we present two convolutional neural networks to estimate the 3D joint positions of a person in a configurable bed from a single pressure image. The first network directly outputs 3D joint positions, while the second outputs a kinematic model that includes estimated joint angles and limb lengths. We evaluated our networks on data from 17 human participants with two bed configurations: supine and seated. Our networks achieved a mean joint position error of 77 mm when tested with data from people outside the training set, outperforming several baselines. We also present a simple mechanical model that provides insight into ambiguity associated with limbs raised off of the pressure mat, and demonstrate that Monte Carlo dropout can be used to estimate pose confidence in these situations. Finally, we provide a demonstration in which a mobile manipulator uses our network's estimated kinematic model to reach a location on a person's body in spite of the person being seated in a bed and covered by a blanket.

During robot-assisted dressing, a robot manipulates a garment in contact with a person's body. Inferring the forces applied to the person's body by the garment might enable a robot to provide more effective assistance and give the robot insight into what the person feels. However, complex mechanics govern the relationship between the robot's end effector and these forces. Using a physics-based simulation and data-driven methods, we demonstrate the feasibility of inferring forces across a person's body using only end effector measurements. Specifically, we present a long short-term memory (LSTM) network that at each time step takes a 9-dimensional input vector of force, torque, and velocity measurements from the robot's end effector and outputs a force map consisting of hundreds of inferred force magnitudes across the person's body. We trained and evaluated LSTMs on two tasks: pulling a hospital gown onto an arm and pulhng shorts onto a leg. For both tasks, the LSTMs produced force maps that were similar to ground truth when visualized as heat maps across the limbs. We also evaluated their performance in terms of root-mean-square error. Their performance degraded when the end effector velocity was increased outside the training range, but generalized well to limb rotations. Overall, our results suggest that robots could learn to infer the forces people feel during robot-assisted dressing, although the extent to which this will generalize to the real world remains an open question.

There is a considerable need for assistive dressing among people with disabilities, and robots have the potential to fulfill this need. However, training such a robot would require extensive trials in order to learn the skills of assistive dressing. Such training would be time-consuming and require considerable effort to recruit participants and conduct trials. In addition, for some cases that might cause injury to the person being dressed, it is impractical and unethical to perform such trials. In this work, we focus on a representative dressing task of pulling the sleeve of a hospital gown onto a person's arm. We present a system that learns a haptic classifier for the outcome of the task given few (2-3) real-world trials with one person. Our system first optimizes the parameters of a physics simulator using real-world data. Using the optimized simulator, the system then simulates more haptic sensory data with noise models that account for randomness in the experiment. We then train hidden Markov Models (HMMs) on the simulated haptic data. The trained HMMs can then be used to classify and predict the outcome of the assistive dressing task based on haptic signals measured by a real robot's end effector. This system achieves 92.83% accuracy in classifying the outcome of the robot-assisted dressing task with people not included in simulation optimization. We compare our classifiers to those trained on real-world data. We show that the classifiers from our system can categorize the dressing task outcomes more accurately than classifiers trained on ten times more real data.

Activities of daily living (ADLs) are important for quality of life. Robotic assistance offers the opportunity for people with disabilities to perform ADLs on their own. However, when a complex semi-autonomous system provides real-world assistance, occasional anomalies are likely to occur. Robots that can detect, classify and respond appropriately to common anomalies have the potential to provide more effective and safer assistance. We introduce a multimodal execution monitor to detect and classify anomalous executions when robots operate near humans. Our system builds on our past work on multimodal anomaly detection. Our new monitor classifies the type and cause of common anomalies using an artificial neural network. We implemented and evaluated our execution monitor in the context of robot-assisted feeding with a general-purpose mobile manipulator. In our evaluations, our monitor outperformed baseline methods from the literature. It succeeded in detecting 12 common anomalies from 8 able-bodied participants with 83% accuracy and classifying the types and causes of the detected anomalies with 90% and 81% accuracies, respectively. We then performed an in-home evaluation with Henry Evans, a person with severe quadriplegia. With our system, Henry successfully fed himself while the monitor detected, classified the types, and classified the causes of anomalies with 86%, 90%, and 54% accuracy, respectively.

Online detection of anomalous execution can be valuable for robot manipulation, enabling robots to operate more safely, determine when a behavior is inappropriate, and otherwise exhibit more common sense. By using multiple complementary sensory modalities, robots could potentially detect a wider variety of anomalies, such as anomalous contact or a loud utterance by a human. However, task variability and the potential for false positives make online anomaly detection challenging, especially for long-duration manipulation behaviors. In this paper, we provide evidence for the value of multimodal execution monitoring and the use of a detection threshold that varies based on the progress of execution. Using a data-driven approach, we train an execution monitor that runs in parallel to a manipulation behavior. Like previous methods for anomaly detection, our method trains a hidden Markov model (HMM) using multimodal observations from non-anomalous executions. In contrast to prior work, our system also uses a detection threshold that changes based on the execution progress. We evaluated our approach with haptic, visual, auditory, and kinematic sensing during a variety of manipulation tasks performed by a PR2 robot. The tasks included pushing doors closed, operating switches, and assisting able-bodied participants with eating yogurt. In our evaluations, our anomaly detection method performed substantially better with multimodal monitoring than single modality monitoring. It also resulted in more desirable ROC curves when compared with other detection threshold methods from the literature, obtaining higher true positive rates for comparable false positive rates.

Robots could benefit from maps that represent haptic properties of their surroundings. By touching locations with tactile sensors, robots can infer haptic properties of their surroundings, but touching all locations would be prohibitive. We present an algorithm that uses touch and vision to efficiently produce a dense haptic map. Our approach assumes that surfaces near a robot that are visually similar are more likely to have similar haptic properties. Given an image and sparse haptic labels, our algorithm uses a dense conditional random field (CRF) to produce a haptic map with labels for all image pixels. In an evaluation using images with idealized haptic labels, our algorithm substantially outperformed a previous algorithm. It also enabled a real robot to label leaves and trunks after reaching into artificial foliage. In addition, we show that our algorithm can use a convolutional neural network (CNN) for material recognition from Bell et al. that we modified and fine-tuned. This CNN provides estimated probabilities for haptic labels using vision alone, which enables the algorithm to infer haptic labels before the robot makes contact with anything. In our evaluation, using this CNN further improved performance.

Robots that effectively manipulate the human body could potentially be useful in a wide variety of applications, including assistive applications for people with disabilities. Toward this end, we present a method to enable robots to compliantly manipulate human limbs. Our approach uses model predictive control (MPC). Given an action by the robot, the model predicts how the human body will move and what forces the robot will apply to the human body. The robot uses this model to optimize its actions to achieve desired motions of the human body while controlling applied forces. This optimization is subject to various constraints, including constraints to avoid hyperextension of the human's joints and to avoid slipping of the robot's end effectors. In this paper, our controller uses a quasistatic model of the human limb in contact with the robot's end effectors, which have linear Cartesian stiffness with respect to Cartesian equilibrium positions. We evaluated our approach in simulation with the specific task of lifting the leg of a human body in a supine position (i.e., lying down). In our tests, we varied the goal configuration for the human leg, the stiffness of the robot's two end effectors, and the model error (i.e., the difference between the controller's model of the human body and the actual human body). Our evaluation demonstrates the feasibility of our approach, since our controller performed well in terms of the forces the robot applied to the human leg and the human leg's motions.

Dressing is an important activity of daily living (ADL) with which many people require assistance due to impairments. Robots have the potential to provide dressing assistance, but physical interactions between clothing and the human body can be complex and difficult to visually observe. We provide evidence that data-driven haptic perception can be used to infer relationships between clothing and the human body during robot-assisted dressing. We conducted a carefully controlled experiment with 12 human participants during which a robot pulled a hospital gown along the length of each person's forearm 30 times. This representative task resulted in one of the following three outcomes: the hand missed the opening to the sleeve; the hand or forearm became caught on the sleeve; or the full forearm successfully entered the sleeve. We found that hidden Markov models (HMMs) using only forces measured at the robot's end effector classified these outcomes with high accuracy. The HMMs' performance generalized well to participants (98.61% accuracy) and velocities (98.61% accuracy) outside of the training data. They also performed well when we limited the force applied by the robot (95.8% accuracy with a 2N threshold), and could predict the outcome early in the process. Despite the lightweight hospital gown, HMMs that used forces in the direction of gravity substantially outperformed those that did not. The best performing HMMs used forces in the direction of motion and the direction of gravity.

We consider the problem of enabling a robot to efficiently obtain a dense haptic map of its visible surroundings using the complementary properties of vision and tactile sensing. Our approach assumes that visible surfaces that look similar to one another are likely to have similar haptic properties. We present an iterative algorithm that enables a robot to infer dense haptic labels across visible surfaces when given a color-plus-depth (RGB-D) image along with a sequence of sparse haptic labels representative of what could be obtained via tactile sensing. Our method uses a color-based similarity measure and connected components on color and depth data. We evaluated our method using several publicly available RGBD image datasets with indoor cluttered scenes pertinent to robot manipulation. We analyzed the effects of algorithm parameters and environment variation, specifically the level of clutter and the type of setting, like a shelf, table top, or sink area. In these trials, the visible surface for each object consisted of an average of 8602 pixels, and we provided the algorithm with a sequence of haptically-labeled pixels up to a maximum of 40 times the number of objects in the image. On average, our algorithm correctly assigned haptic labels to 76.02% of all of the object pixels in the image given this full sequence of labels. We also performed experiments with the humanoid robot DARCI reaching in a cluttered foliage environment while using our algorithm to create a haptic map. Doing so enabled the robot to reach goal locations using a single plan after a single greedy reach, while our previous tactile-only mapping method required 5 or more plans to reach each goal.

When a mobile manipulator functions as an assistive device, the robot's initial configuration and the configuration of the environment can impact the robot's ability to provide effective assistance. Selecting initial configurations for assistive tasks can be challenging due to the high number of degrees of freedom of the robot, the environment, and the person, as well as the complexity of the task. In addition, rapid selection of initial conditions can be important, so that the system will be responsive to the user and will not require the user to wait a long time while the robot makes a decision. To address these challenges, we present Task-centric initial Configuration Selection (TCS), which unlike previous work uses a measure of task-centric manipulability to accommodate state estimation error, considers various environmental degrees of freedom, and can find a set of configurations from which a robot can perform a task. TCS performs substantial offline computation, so that it can rapidly provide solutions at run time. At run time, the system performs an optimization over candidate initial configurations using a utility function that can include factors such as movement costs for the robot's mobile base. To evaluate TCS, we created models of 11 activities of daily living (ADLs) and evaluated TCS's performance with these 11 assistive tasks in a computer simulation of a PR2, a robotic bed, and a model of a human body. TCS performed as well or better than a baseline algorithm in all of our tests against state estimation error.

Robots are being designed to assist older adults in their homes. However, we lack a clear understanding of the aspects of robot appearance older adults pay attention to and consider important for accepting a robot in their homes. The goal of this study was to systematically assess older adults' reactions to a specific robot's appearance in the context of home use. Independent living older adults interacted with the PR2 and were interviewed about its appearance. In general, there was expectation for a small sized robot that fits with the home and is easy to control. There was variability for desired human-likeness, although some human-like characteristics might be acceptable to most.

We address the challenge of finding and navigating to an object with an attached ultra-high frequency radio-frequency identification (UHF RFID) tag. With current off-the-shelf technology, one can affix inexpensive self-adhesive UHF RFID tags to hundreds of objects, thereby enabling a robot to sense the RF signal strength it receives from each uniquely identified object. The received signal strength indicator (RSSI) associated with a tagged object varies widely and depends on many factors, including the object's pose, material properties and surroundings. This complexity creates challenges for methods that attempt to explicitly estimate the object's pose. We present an alternative approach that formulates finding and navigating to a tagged object as an optimization problem where the robot must find a pose of a directional antenna that maximizes the RSSI associated with the target tag. We then present three autonomous robot behaviors that together perform this optimization by combining global and local search. The first behavior uses sparse sampling of RSSI across the entire environment to move the robot to a location near the tag; the second samples RSSI over orientation to point the robot toward the tag; and the third samples RSSI from two antennas pointing in different directions to enable the robot to approach the tag. We justify our formulation using the radar equation and associated literature. We also demonstrate that it has good performance in practice via tests with a PR2 robot from Willow Garage in a house with a variety of tagged household objects.

Often in highly-cluttered environments, a robot can observe the exterior of the environment with ease, but cannot directly view nor easily infer its detailed internal structure (e.g., dense foliage or a full refrigerator shelf). We present a data-driven approach that greatly improves a robot's success at reaching to a goal location in the unknown interior of an environment based on observable external properties, such as the category of the clutter and the locations of openings into the clutter (i.e., apertures). We focus on the problem of selecting a good initial configuration for a manipulator when reaching with a greedy controller. We use density estimation to model the probability of a successful reach given an initial condition and then perform constrained optimization to find an initial condition with the highest estimated probability of success. We evaluate our approach with two simulated robots reaching in clutter, and provide a demonstration with a real PR2 robot reaching to locations through random apertures. In our evaluations, our approach significantly outperformed two alternative approaches when making two consecutive reach attempts to goals in distinct categories of unknown clutter. Our approach only uses sparse readily-apparent features.

Biological organisms benefit from tactile sensing across the entire surfaces of their bodies. Robots may also be able to benefit from this type of sensing, but fully covering a robot with robust and capable tactile sensors entails numerous challenges. To date, most tactile sensors for robots have been used to cover rigid surfaces. In this paper, we focus on the challenge of tactile sensing across articulated joints, which requires sensing across a surface whose geometry varies over time. We first demonstrate the importance of sensing across joints by simulating a planar arm reaching in clutter and finding the frequency of contact at the joints. We then present a simple model of how much a tactile sensor would need to stretch in order to cover a 2 degree-of-freedom (DoF) wrist joint. Next, we describe and characterize a new tactile sensor made with stretchable fabrics. Finally, we present results for a stretchable sleeve with 25 tactile sensors that covers the forearm, 2 DoF wrist, and end effector of a humanoid robot. This sleeve enabled the robot to reach a target in instrumented clutter and reduce contact forces.

We present a unique multi-antenna RFID reader (a sensor) embedded in a robot's manipulator that is designed to operate with ordinary UHF RFID tags in a short-range, near-field electromagnetic regime. Using specially designed near-field antennas enables our sensor to obtain spatial information from tags at ranges of less than 1 meter. In this work, we characterize the near-field sensor's ability to detect tagged objects in the robots manipulator, present robot behaviors to determine the identity of a grasped object, and investigate how additional RF signal properties can be used for “pre-touch” capabilities such as servoing to grasp an object. The future combination of long-range (far-field) and short-range (near-field) UHF RFID sensing has the potential to enable roboticists to jump-start applications by obviating or supplementing false-positive-prone visual object recognition. These techniques may be especially useful in the healthcare and service sectors, where mis-identification of an object (for example, a medication bottle) could have catastrophic consequences.

We introduce ROS Commander (ROSCo), an open source system that enables expert users to construct, share, and deploy robot behaviors for home robots. A user builds a behavior in the form of a Hierarchical Finite State Machine (HFSM) out of generic, parameterized building blocks, with a real robot in the develop and test loop. Once constructed, users save behaviors in an open format for direct use with robots, or for use as parts of new behaviors. When the system is deployed, a user can show the robot where to apply behaviors relative to fiducial markers (AR Tags), which allows the robot to quickly become operational in a new environment. We show evidence that the underlying state machine representation and current building blocks are capable of spanning a variety of desirable behaviors for home robots, such as opening a refrigerator door with two arms, flipping a light switch, unlocking a door, and handing an object to someone. Our experiments show that sensor-driven behaviors constructed with ROSCo can be executed in realistic home environments with success rates between 80% and 100%. We conclude by describing a test in the home of a person with quadriplegia, in which the person was able to automate parts of his home using previously-built behaviors.

Many older adults wish to remain in their own homes as they age [16]. However, challenges in performing home upkeep tasks threaten an older adult's ability to age in place. Even healthy independently living older adults experience challenges in maintaining their home [13]. Challenges with home tasks can be compensated through technology, such as home robots. However, for home robots to be adopted by older adult users, they must be designed to meet older adults' needs for assistance and the older users must be amenable to robot assistance for those needs. We conducted a needs assessment to (1) assess older adults' openness to assistance from robots; and (2) understand older adults' opinions about using an assistive robot to help around the home. We administered questionnaires and conducted structured group interviews with 21 independently living older adults (ages 65-93). The questionnaire data suggest that older adults prefer robot assistance for cleaning and fetching/organizing tasks overall. However their assistance preferences discriminated between tasks. The interview data provided insight as to why they hold such preferences. Older adults reported benefits of robot assistance (e.g., the robot compensating for limitations, saving them time and effort, completing undesirable tasks, and performing tasks at a high level of performance). Participants also reported concerns such as the robot damaging the environment, being unreliable at or incapable of doing a task, doing tasks the older adult would rather do, or taking up too much space/storage. These data, along with specific comments from participant interviews, provide the basis for preliminary recommendations for designing mobile manipulator robots to support aging in place.

In this paper, we demonstrate data-driven inference of mechanical properties of objects using a tactile sensor array (skin) covering a robot's forearm. We focus on the mobility (sliding vs. fixed), compliance (soft vs. hard), and identity of objects in the environment, as this information could be useful for efficient manipulation and search. By using the large surface area of the forearm, a robot could potentially search and map a cluttered volume more efficiently, and be informed by incidental contact during other manipulation tasks. Our approach tracks a contact region on the forearm over time in order to generate time series of select features, such as the maximum force, contact area, and contact motion. We then process and reduce the dimensionality of these time series to generate a feature vector to characterize the contact. Finally, we use the k-nearest neighbor algorithm (k-NN) to classify a new feature vector based on a set of previously collected feature vectors. Our results show a high cross-validation accuracy in both classification of mechanical properties and object recognition. In addition, we analyze the effect of taxel resolution, duration of observation, feature selection, and feature scaling on the classification accuracy.

The Robots for Humanity project aims to enable people with severe motor impairments to interact with their own bodies and their environment through the use of an assistive mobile manipulator, thereby improving their quality of life. Assistive mobile manipulators (AMMs) are mobile robots that physically manipulate the world in order to provide assistance to people with disabilities. They present an exciting frontier for assistive technology, as they can operate away from the user, have a large dexterous workspace (due to their mobility), and not directly encumber their users. The cornerstone of this project is an ongoing, interactive design process with a quadriplegic user, Henry Evans, and his wife and primary caregiver, Jane Evans. Henry has been enabled, through the use of a PR2 robot, to scratch his own face, shave, fetch a towel from his kitchen, and hand out Halloween candy to trick-ortreating children at a local mall.

Persons with severe motor impairments depend heavily upon caregivers for the performance of everyday tasks. Ongoing work is exploring the potential of giving motor-impaired users control of semi-autonomous assistive mobile manipulators to enable them to perform some self-care tasks such as scratching or shaving. Because these users are less able to escape a robot malfunction, or operate a traditional run-stop, physical human-robot interaction poses safety risks. We review approaches to safety in assistive robotics with a focus on accessible run-stops, and propose wincing as an accessible gesture for activating a run-stop device. We also present the wouse, a novel device for detecting wincing from skin movement near the eye, consisting of optical mouse components mounted near a user's temple via safety goggles. Using this device, we demonstrate a complete system to run-stop a Willow Garage PR2 robot, and perform two preliminary user studies. The first study examines discrimination of wincing from self-produced facial expressions. The results indicate the possibility for discrimination, though variability between users and inconsistent detection of skin movement remain significant challenges. The second experiment examines discrimination of wincing from external mechanical manipulations of the face during self-care tasks. The results indicate that the wouse, using a classifier trained with data from the first experiment, can be used during face-manipulation tasks. The device produced no false positives, but succeeded in correctly identifying wincing events in only two of four subjects.

Hygiene and feeding are activities of daily living (ADLs) that often involve contact with a person's face. Robots can assist people with motor impairments to perform these tasks by holding a tool that makes contact with the care receiver's face. By sensing the forces applied to the face with the tool, robots could potentially provide assistance that is more comfortable, safe, and effective. In order to inform the design of robotic controllers and assistive robots, we investigated the forces able-bodied people apply to themselves when wiping and shaving their faces. We present our methods for capturing and modeling these forces, results from a study with 9 participants, and recommendations for assistive robots. Our contributions include a trapezoidal force model that assumes participants have a target force they attempt to achieve for each stroke of the tool. We discuss advantages of this 3 parameter model and show that it fits our data well relative to other candidate models. We also provide statistics of the models' rise rates, fall rates, and target forces for the 9 participants in our study. In addition, we illustrate how the target forces varied based on the task, participant, and location on the face.

By initiating physical contact with people, robots can be more useful. For example, a robotic caregiver might make contact to provide physical assistance or facilitate communication. So as to better understand how people respond to robot-initiated touch, we conducted a 2x2 between-subjects experiment with 56 people in which a robotic nurse autonomously touched and wiped the subject's forearm. Our independent variables were whether or not the robot verbally warned the person before contact, and whether the robot verbally indicated that the touch was intended to clean the person's skin (instrumental touch) or to provide comfort (affective touch). On average, regardless of the treatment, participants had a generally positive subjective response. However, with instrumental touch people responded significantly more favorably. Since the physical behavior of the robot was the same for all trials, our results demonstrate that the perceived intent of the robot can significantly influence a person's subjective response to robot-initiated touch. Our results suggest that roboticists should consider this factor in addition to the mechanics of physical interaction. Unexpectedly, we found that participants tended to respond more favorably without a verbal warning. Although inconclusive, our results suggest that verbal warnings prior to contact should be carefully designed, if used at all.

When a user is in close proximity to a robot, physical contact becomes a potentially valuable channel for communication. People often use direct physical contact to guide a person to a desired location (e.g., leading a child by the hand) or to adjust a person's posture for a task (e.g., a dance instructor working with a dancer). Within this paper, we present an implementation and evaluation of a direct physical interface for a human-scale anthropomorphic robot. We define a direct physical interface (DPI) to be an interface that enables a user to influence a robot's behavior by making contact with its body. Human-human interaction inspired our interface design, which enables a user to lead our robot by the hand and position its arms. We evaluated this interface in the context of assisting nurses with patient lifting, which we expect to be a high-impact application area. Our evaluation consisted of a controlled laboratory experiment with 18 nurses from the Atlanta area of Georgia, USA. We found that our DPI significantly outperformed a comparable wireless gamepad interface in both objective and subjective measures, including number of collisions, time to complete the tasks, workload (Raw Task Load Index), and overall preference. In contrast, we found no significant difference between the two interfaces with respect to the users' perceptions of personal safety.

Previously, we have presented an implementation of impedance control inspired by the Equilibrium Point Hypothesis that we refer to as equilibrium point control (EPC). We have demonstrated that EPC can enable a robot in a fixed position to robustly pull open a variety of doors and drawers, and infer their kinematics without detailed prior models. In this paper, we extend this framework to support autonomous motion of the robot's omni-directional base both before and during pulling. With our new methods, we show that the robot can autonomously approach and open doors and drawers for which only the location and orientation of the handle have been provided. We also demonstrate that EPC can coordinate the movement of the robot's omni-directional base and compliant arm while pulling open a door or drawer, which leads to significantly improved performance. Through 40 trials with 10 different doors and drawers, we empirically demonstrated the robustness of the system. The robot succeeded in 37 out of 40 trials, and had no more than a single failure for any particular door or drawer.

An omnidirectional Mecanum base allows for more flexible mobile manipulation. However, slipping of the Mecanum wheels results in poor dead-reckoning estimates from wheel encoders, limiting the accuracy and overall utility of this type of base. We present a system with a downward-facing camera and light ring to provide robust visual odometry estimates. We mounted the system under the robot which allows it to operate in conditions such as large crowds or low ambient lighting. We demonstrate that the visual odometry estimates are sufficient to generate closed-loop PID (Proportional Integral Derivative) and LQR (Linear Quadratic Regulator) controllers for motion control in three different scenarios: waypoint tracking, small disturbance rejection, and sideways motion. We report quantitative measurements that demonstrate superior control performance when using visual odometry compared to wheel encoders. Finally, we show that this system provides high-fidelity odometry estimates and is able to compensate for wheel slip on a four-wheeled omnidirectional mobile robot base.

This paper describes the design and implementation of a behavior that allows a robot with a compliant arm to perform wiping motions that are involved in bed baths. A laser-based operator-selection interface enables an operator to select an area to clean, and the robot autonomously performs a wiping motion using equilibrium point control. We evaluated the performance of the system by measuring the ability of the robot to remove an area of debris on human skin. We tested the performance of the behavior algorithm by commanding the robot to wipe off a 1-inch square area of debris placed on the surface of the upper arm, forearm, thigh, and shank of a human subject. Using image processing, we determined the hue content of the debris and used this representation to determine the percentage of debris that remained on the arm after the robot completed the task. In our experiments, the robot removed most of the debris (>96%) on four parts of the limbs. In addition, the robot performed the wiping task using relatively low force (<;3 N).

Many tasks that would be of benefit to users in domestic environments require that robots manipulate articulated objects such as doors and drawers. In this paper, we present a novel approach that simultaneously estimates the kinematic model of an articulated object based on the trajectory described by the robot's end effector, and uses this model to predict the future trajectory of the end effector. One advantage of our approach is that the robot can directly use these predictions to generate an equilibrium point control path for operating the mechanism. Additionally, our approach can improve these predictions based on previously learned articulation models. We have implemented and tested our approach on a real mobile manipulator. Through 40 trials, we show that the robot can reliably open various household objects, including cabinet doors, sliding doors, office drawers, and a dishwasher. Furthermore, we demonstrate that using the information from previous interactions as a prior significantly improves the prediction accuracy.

When young children play, they often manipulate toys that have been specifically designed to accommodate and stimulate their perceptual-motor skills. Robotic playmates capable of physically manipulating toys have the potential to engage children in therapeutic play and augment the beneficial interactions provided by overtaxed care givers and costly therapists. To date, assistive robots for children have almost exclusively focused on social interactions and teleoperative control. Within this paper we present progress towards the creation of robots that can engage children in manipulative play. First, we present results from a survey of popular toys for children under the age of 2 which indicates that these toys share simplified appearance properties and are designed to support a relatively small set of coarse manipulation behaviors. We then present a robotic control system that autonomously manipulates several toys by taking advantage of this consistent structure. Finally, we show results from an integrated robotic system that imitates visually observed toy playing activities and is suggestive of opportunities for robots that play with toys.

Unstructured, human environments present great challenges and opportunities for robotic manipulation and grasping. Robots that reliably grasp household objects with unknown or uncertain properties would be especially useful, since these robots could better generalize their capabilities across the wide variety of objects found within domestic environments. Within this paper, we address the problem of picking up an object sitting on a plane in isolation, as can occur when someone drops an object on the floor - a common problem for motor- impaired individuals. We assume that the robot has the ability to coarsely position itself in front of the object, but otherwise grasps the object with an open-loop strategy that does not vary from object to object. We present a novel end effector that is capable of robustly picking up a diverse array of everyday handheld objects given these conditions. This straight-forward, inexpensive, nonpre- hensile end effector combines a compliant finger with a thin planar component with a leading wedge that slides underneath the object. We empirically validated the efficacy of this design through a set of 1096 trials over which we systematically varied the object location, object type, object configuration, and floor characteristics. Our implementation, which we mounted on a iRobot Create, had a success rate of 94.71 % on 680 trials, which used 4 floor types with 34 objects of particular relevance to assistive applications in 5 different poses each (4x34x5=680). The robot also had strong performance with objects that would be difficult to grasp using a traditional end effector, such as a dollar bill, a pill, a cloth, a credit card, a coin, keys, and a watch. Prior to this test, we performed 416 trials in order to assess the performance of the end effector with respect to variations in object position.

In this work we present a set of integrated methods that enable an RFID-enabled mobile manipulator to approach and grasp an object to which a self-adhesive passive (battery-free) UHF RFID tag has been affixed. Our primary contribution is a new mode of perception that produces images of the spatial distribution of received signal strength indication (RSSI) for each of the tagged objects in an environment. The intensity of each pixel in the 'RSSI image' is the measured RF signal strength for a particular tag in the corresponding direction. We construct these RSSI images by panning and tilting an RFID reader antenna while measuring the RSSI value at each bearing. Additionally, we present a framework for estimating a tagged object's 3D location using fused ID-specific features derived from an RSSI image, a camera image, and a laser range finder scan. We evaluate these methods using a robot with actuated, long-range RFID antennas and finger-mounted short-range antennas. The robot first scans its environment to discover which tagged objects are within range, creates a user interface, orients toward the user-selected object using RF signal strength, estimates the 3D location of the object using an RSSI image with sensor fusion, approaches and grasps the object, and uses its finger-mounted antennas to confirm that the desired object has been grasped. In our tests, the sensor fusion system with an RSSI image correctly located the requested object in 17 out of 18 trials (94.4%), an 11.1% improvement over the system's performance when not using an RSSI image. The robot correctly oriented to the requested object in 8 out of 9 trials (88.9%), and in 3 out of 3 trials the entire system successfully grasped the object selected by the user.

Delivering an object to a user would be a generally useful capability for service robots. Within this paper, we look at this capability in the context of assistive object retrieval for motor-impaired users. We first describe a behavior-based system that enables our mobile robot EL-E to autonomously deliver an object to a motor-impaired user. We then present our evaluation of this system with 8 motor-impaired patients from the Emory ALS Center. As part of this study, we compared handing the object to the user (direct delivery) with placing the object on a nearby table (indirect delivery). We tested the robot delivering a cordless phone, a medicine bottle, and a TV remote, which were ranked as three of the top four most important objects for robotic delivery by ALS patients in a previous study. Overall, the robot successfully delivered these objects in 126 out of 144 trials (88%) with a success rate of 97% for indirect delivery and 78% for direct delivery. In an accompanying survey, participants showed high satisfaction with the robot with 4 people preferring direct delivery and 4 people preferring indirect delivery. Our results indicate that indirect delivery to a surface can be a robust and reliable delivery method with high user satisfaction, and that robust direct delivery will require methods that handle diverse postures and body types.

El-E ("Ellie") is a prototype assistive robot designed to help people with severe motor impairments manipulate everyday objects. When given a 3D location, El-E can autonomously approach the location and pick up a nearby object. Based on interviews of patients with amyotrophic lateral sclerosis (ALS), we have developed and tested three distinct interfaces that enable a user to provide a 3D location to El-E and thereby select an object to be manipulated: an ear-mounted laser pointer, a hand-held laser pointer, and a touch screen interface. Within this paper, we present the results from a user study comparing these three user interfaces with a total of 134 trials involving eight patients with varying levels of impairment recruited from the Emory ALS Clinic. During this study, participants used the three interfaces to select everyday objects to be approached, grasped, and lifted off of the ground.

We present a novel interface for human-robot interaction that enables a human to intuitively and unambiguously select a 3D location in the world and communicate it to a mobile robot. The human points at a location of interest and illuminates it (``clicks it'') with an unaltered, off-the-shelf, green laser pointer. The robot detects the resulting laser spot with an omnidirectional, catadioptric camera with a narrow-band green filter. After detection, the robot moves its stereo pan/tilt camera to look at this location and estimates the location's 3D position with respect to the robot's frame of reference.

We present a novel antenna and system architecture for mobile manipulation based on passive RFID technology operating in the 850 MHz - 950 MHz ultra-high-frequency (UHF) spectrum. This system exploits the electromagnetic properties of UHF radio signals to present a mobile robot with both wide-angle dasiaperipheral visionpsila, sensing multiple tagged objects in the area in front of the robot, and focused, high-acuity dasiacentral visionpsila, sensing only tagged objects close to the end effector of the manipulator. These disparate tasks are performed using the same UHF RFID tag, coupled in two different electromagnetic modes. Wide-angle sensing is performed with an antenna designed for far-field electromagnetic wave propagation, while focused sensing is performed with a specially designed antenna mounted on the end effector that optimizes near-field magnetic coupling. We refer to this RFID system as dasiafoveatedpsila, by analogy with the anatomy of the human eye. We report a series of experiments on an untethered autonomous mobile manipulator in a 2.5D environment that demonstrate the features of this architecture using two novel behaviors, one in which data from the far-field antenna is used to determine if a specific tagged object is present in the robotpsilas working area and to navigate to that object, and a second using data from the near-field antenna to grasp a specified object from a collection of visually identical objects. The same UHF RFID tag is used to facilitate both the navigation and grasping tasks.

We present a novel particle filter implementation for estimating the pose of tags in the environment with respect to an RFID-equipped robot. This particle filter combines signals from a specially designed RFID antenna system with odometry and an RFID signal propagation model. Our model includes antenna characteristics, direct-path RF propagation, and multipath RF propagation. We first describe a novel 6-antenna RFID sensor system that provides the robot with a 360-degree view of the tags in its environment. We then present the results of real-world evaluation where RFID-inferred tag position is compared with ground truth data from a laser range-finder. In our experiments the system is shown to estimate the pose of UHF RFID tags in a real-world environment without requiring a priori training or map-building. The system exhibits 6.1 deg mean bearing error and 0.69 m mean range error over robot to tag distances of over 4 m in an environment with significant multipath. The RFID system provides the ability to uniquely identify specific tagged locations and objects, and to discriminate among multiple tagged objects in the field at the same time, which are important capabilities that a laser range-finder does not provide. We expect that this new type of multiple-antenna RFID system, including particle filters that incorporate RF signal propagation models, will prove to be a valuable sensor for mobile robots operating in semi-structured environments where RFID tags are present.

We present a new behavior selection system for human-robot interaction that maps virtual buttons overlaid on the physical environment to the robotpsilas behaviors, thereby creating a clickable world. The user clicks on a virtual button and activates the associated behavior by briefly illuminating a corresponding 3D location with an off-the-shelf green laser pointer. As we have described in previous work, the robot can detect this click and estimate its 3D location using an omnidirectional camera and a pan/tilt stereo camera. In this paper, we show that the robot can select the appropriate behavior to execute using the 3D location of the click, the context around this 3D location, and its own state. For this work, the robot performs this selection process using a cascade of classifiers. We demonstrate the efficacy of this approach with an assistive object-fetching application. Through empirical evaluation, we show that the 3D location of the click, the state of the robot, and the surrounding context is sufficient for the robot to choose the correct behavior from a set of behaviors and perform the following tasks: pick-up a designated object from a floor or table, deliver an object to a designated person, place an object on a designated table, go to a designated location, and touch a designated location with its end effector.

In this paper, we discuss a methodology to extract play primitives, defined as a sequence of low-level motion behaviors identified during a playing action, such as stacking or inserting a toy. Our premise is that if a robot could interpret the basic movements of a humanpsilas play, it will be able to interact with many different kinds of toys, in conjunction with its human playmate. As such, we present a method that combines motion behavior analysis and behavior sequencing, which capitalizes on the inherent characteristics found in the dynamics of play such as the limited domain of the objects and manipulation skills required. In this paper, we give details on the approach and present results from applying the methodology to a number of play scenarios.

For manipulation tasks, the transfer of objects between humans and robots is a fundamental way to coordinate activity and cooperatively perform useful work. Within this paper we demonstrate that robots and people can effectively and intuitively work together by directly handing objects to one another. First, we present experimental results that demonstrate that subjects without explicit instructions or robotics expertise can successfully hand objects to a robot and take objects from a robot in response to reaching gestures. Moreover, when handing an object to the robot, subjects control the object's position and orientation to match the configuration of the robot's hand, thereby simplifying robotic grasping and offering opportunities to simplify the manipulation task. Second, we present a robotic application that relies on this form of human-robot interaction. This application enables a humanoid robot to help a user place objects on a shelf, perform bimanual insertion tasks, and hold a box within which the user can place objects. By handing appropriate objects to the robot, the human directly and intuitively controls the robot. Through this interaction, the human and robot complement one another's abilities and work together to achieve results.