Despite the importance of sensitive skin for living creatures, most robots can feel contact on only a tiny fraction of their exterior, if at all. Furthermore, typical robot reactions to touch are limited to event-based acknowledgments, lacking perceptual richness, lifelike positive/negative responses, and temporal dynamics. We address these gaps by introducing a practical full-body tactile-perception system for social robots, turning a NAO robot into the Haptic Empathetic Robot Animal (HERA). The sixteen main regions of the robot's body are instrumented with soft resistive tactile sensors covered by a tailored koala suit. Windows of each time-varying sensor output are continually classified into five gestures at two intensities via a two-stage machine-learning model. On challenging testing data containing simultaneous contacts, touch detection achieves an F1 score of 0.773, and gesture recognition achieves 52.2% accuracy (5.2 times chance); considering the temporal, spatial, and semantic adjacency of the applied touches increases these metrics to 0.896 and 86.6%, respectively. In turn, each detected contact drives a real-time emotion model that represents the robot's affective state as a second-order dynamic system analogous to a mass-spring-damper. This model's parameters control the robot's disposition, stoicism, and calmness. We explain the connections between HERA's hardware and software subsystems and demonstrate their combined ability to create an affective robot that feels both touch and emotion.

Humanoid social robots need to be able to move expressively. Traditional manipulation-focused teleoperation systems primarily control the end-effector's position and orientation, neglecting the extra degrees of freedom in human and robotic arms, which can lead to unnatural movements. This demonstration presents our Optimization-based Customizable Retargeting Algorithm (OCRA), designed for real-time motion mapping between dissimilar kinematic chains. OCRA functions well with widely varying robot-arm joint configurations. The presenter will use a commercial motion-capture suit to teleoperate the upper body of a NAO humanoid robot, demonstrating OCRA's ability to create intuitive, human-like movements in real time.

Capturing accurate 3D human pose in the wild would provide valuable data for training pose estimation and motion generation methods. While video-based estimation approaches have become increasingly accurate, they often fail in common scenarios involving self-contact, such as a hand touching the face. In contrast, wearable bioimpedance sensing can cheaply and unobtrusively measure ground-truth skin-to-skin contact. Consequently, we propose a novel framework that combines visual pose estimators with bioimpedance sensing to capture the 3D pose of people by taking self-contact into account. Our method, BioTUCH, initializes the pose using an off-the-shelf estimator and introduces contact-aware pose optimization during measured self-contact: reprojection error and deviations from the input estimate are minimized while enforcing vertex proximity constraints. We validate our approach using a new dataset of synchronized RGB video, bioimpedance measurements, and 3D motion capture. Testing with three input pose estimators, we demonstrate an average of 11.7% improvement in reconstruction accuracy. We also present a miniature wearable bioimpedance sensor that enables efficient large-scale collection of contact-aware training data for improving pose estimation and generation using BioTUCH. Code and data are available at biotuch.is.tue.mpg.de

Accurate 3D pose estimation of grasped objects is an important prerequisite for robots to perform assembly or in-hand manipulation tasks, but object occlusion by the robot's own hand greatly increases the difficulty of this perceptual task. Here, we propose that combining visual information and proprioception with binary, low-resolution tactile contact measurements from across the interior surface of an articulated robotic hand can mitigate this issue. The visuo-tactile object-pose-estimation problem is formulated probabilistically in a factor graph. The pose of the object is optimized to align with the three kinds of measurements using a robust cost function to reduce the influence of visual or tactile outlier readings. The advantages of the proposed approach are first demonstrated in simulation: a custom 15-DoF robot hand with one binary tactile sensor per link grasps 17 YCB objects while observed by an RGB-D camera. This low-resolution inhand tactile sensing significantly improves object-pose estimates under high occlusion and also high visual noise. We also show these benefits through grasping tests with a preliminary real version of our tactile hand, obtaining reasonable visuo-tactile estimates of object pose at approximately 13.3 Hz on average.

Approximating model predictive control (MPC) using imitation learning (IL) allows for fast control without solving expensive optimization problems online. However, methods that use neural networks in a simple L2-regression setup fail to approximate multi-modal (set-valued) solution distributions caused by local optima found by the numerical solver or nonconvex constraints, such as obstacles, significantly limiting the applicability of approximate MPC in practice. We solve this issue by using diffusion models to accurately represent the complete solution distribution (i.e., all modes) up to kilohertz sampling rates. This work shows that diffusion-based AMPC significantly outperforms L2-regression-based approximate MPC for multi-modal action distributions. In contrast to most earlier work on IL, we also focus on running the diffusion-based controller at a higher rate and in joint space instead of end-effector space. Additionally, we propose the use of gradient guidance during the denoising process to consistently pick the same mode in closed loop to prevent switching between solutions. We propose using the cost and constraint satisfaction of the original MPC problem during parallel sampling of solutions from the diffusion model to pick a better mode online. We evaluate our method on the fast and accurate control of a 7-DoF robot manipulator both in simulation and on hardware deployed at 250 Hz, achieving a speedup of more than 70 times compared to solving the MPC problem online and also outperforming the numerical optimization (used for training) in success ratio.

Recent efforts to connect builders to digital designs during construction have primarily focused on visual augmented reality, which requires accurate registration and specific lighting, and which could prevent a user from noticing safety hazards. Haptic interfaces, on the other hand, can convey physical design parameters through tangible local cues that don’t distract from the surroundings. We propose two edge-changing haptic devices that use small inertial measurement units (IMUs) and linear actuators to guide users to perform construction tasks in real time: Drangle gives feedback for angling a drill relative to gravity, and Brangle assists with orienting bricks in the plane. We conducted a study with 18 participants to evaluate user performance and gather qualitative feedback. All users understood the edge-changing cues from both devices with minimal training. Drilling holes with Drangle was somewhat less accurate but much faster and easier than with a mechanical guide; 89% of participants preferred Drangle over the mechanical guide. Users generally understood Brangle’s feedback but found its hand-size-specific grip, palmar contact, and attractive tactile cues less intuitive than Drangle’s generalized form factor, fingertip contact, and repulsive cues. After summarizing design considerations, we propose application scenarios and speculate how such devices could improve construction workflows.

Machine learning and deep learning have been used extensively to classify physical surfaces through images and time-series contact data. However, these methods rely on human expertise and entail the time-consuming processes of data and parameter tuning. To overcome these challenges, we propose an easily implemented framework that can directly handle heterogeneous data sources for classification tasks. Our data-versus-data approach automatically quantifies distinctive differences in distributions in a high-dimensional space via kernel two-sample testing between two sets extracted from multimodal data (e.g., images, sounds, haptic signals). We demonstrate the effectiveness of our technique by benchmarking against expertly engineered classifiers for visual-audio-haptic surface recognition due to the industrial relevance, difficulty, and competitive baselines of this application; ablation studies confirm the utility of key components of our pipeline. As shown in our open-source code, we achieve 97.2% accuracy on a standard multi-user dataset with 108 surface classes, outperforming the state-of-the-art machine-learning algorithm by 6% on a more difficult version of the task. The fact that our classifier obtains this performance with minimal data processing in the standard algorithm setting reinforces the powerful nature of kernel methods for learning to recognize complex patterns.Note to Practitioners—We demonstrate how to apply the kernel two-sample test to a surface-recognition task, discuss opportunities for improvement, and explain how to use this framework for other classification problems with similar properties. Automating surface recognition could benefit both surface inspection and robot manipulation. Our algorithm quantifies class similarity and therefore outputs an ordered list of similar surfaces. This technique is well suited for quality assurance and documentation of newly received materials or newly manufactured parts. More generally, our automated classification pipeline can handle heterogeneous data sources including images and high-frequency time-series measurements of vibrations, forces and other physical signals. As our approach circumvents the time-consuming process of feature engineering, both experts and non-experts can use it to achieve high-accuracy classification. It is particularly appealing for new problems without existing models and heuristics. In addition to strong theoretical properties, the algorithm is straightforward to use in practice since it requires only kernel evaluations. Its transparent architecture can provide fast insights into the given use case under different sensing combinations without costly optimization. Practitioners can also use our procedure to obtain the minimum data-acquisition time for independent time-series data from new sensor recordings.

Sign language (SL) is the primary method of communication for the 70 million Deaf people around the world. Video dictionaries of isolated signs are a core SL learning tool. Replacing these with 3D avatars can aid learning and enable AR/VR applications, improving access to technology and online media. However, little work has attempted to estimate expressive 3D avatars from SL video; occlusion, noise, and motion blur make this task difficult. We address this by introducing novel linguistic priors that are universally applicable to SL and provide constraints on 3D hand pose that help resolve ambiguities within isolated signs. Our method, SGNify, captures fine-grained hand pose, facial expression, and body movement fully automatically from in-the-wild monocular SL videos. We evaluate SGNify quantitatively by using a commercial motion-capture system to compute 3D avatars synchronized with monocular video. SGNify outperforms state-of-the-art 3D body-pose-and shape-estimation methods on SL videos. A perceptual study shows that SGNify's 3D reconstructions are significantly more comprehensible and natural than those of previous methods and are on par with the source videos. Code and data are available at sgnify.is.tue.mpg.de.

Robots are increasingly being developed as assistants for household, education, therapy, and care settings. Such robots can use adaptive emotional behavior to communicate warmly and effectively with their users and to encourage interest in extended interactions. However, autonomous physical robots often lack a dynamic internal emotional state, instead displaying brief, fixed emotion routines to promote specific user interactions. Furthermore, despite the importance of social touch in human communication, most commercially available robots have limited touch sensing, if any at all. We propose that users’ perceptions of a social robotic system will improve when the robot provides emotional responses on both shorter and longer time scales (reactions and moods), based on touch inputs from the user. We evaluated this proposal through an online study in which 51 diverse participants watched nine randomly ordered videos (a three-by-three full-factorial design) of the koala-like robot HERA being touched by a human. Users provided the highest ratings in terms of agency, ambient activity, enjoyability, and touch perceptivity for scenarios in which HERA showed emotional reactions and either neutral or emotional moods in response to social touch gestures. Furthermore, we summarize key qualitative findings about users’ preferences for reaction timing, the ability of robot mood to show persisting memory, and perception of neutral behaviors as a curious or self-aware robot.

Electrical resistance tomography (ERT) can be used to create large-scale soft tactile sensors that are flexible and robust. Good performance requires a fast and accurate mapping from the sensor’s sequential voltage measurements to the distribution of force across its surface. However, particularly with multiple contacts, this task is challenging for both previously developed approaches: physics-based modeling and end-to-end data-driven learning. Some promising results were recently achieved using sim-to-real transfer learning, but estimating multiple contact locations and accurate contact forces remains difficult because simulations tend to be less accurate with a high number of contact locations and/or high force. This paper introduces a modular hybrid method that combines simulation data synthesized from an electromechanical finite element model with real measurements collected from a new ERT-based tactile sensor. We use about 290 000 simulated and 90000 real measurements to train two deep neural networks: the first (Transfer-Net) captures the inevitable gap between simulation and reality, and the second (Recon-Net) reconstructs contact forces from voltage measurements. The number of contacts, contact locations, force magnitudes, and contact diameters are evaluated for a manually collected multi-contact dataset of 150 measurements. Our modular pipeline’s results outperform predictions by both a physics-based model and end-to-end learning. Note to Practitioners–ERT-based tactile sensors use high-speed voltage measurements from electrodes distributed over a piezoresistive area to output a force map that shows where contact is occurring, and how strong each contact is. Such sensors hold promise for giving robots and other devices a sense of touch over large surfaces with low hardware complexity. However, the software problem of converting voltages to an accurate force map has not previously been solved well, requiring either extensive model calibration or extensive contact data collection. This paper suggests a hybrid approach where a straightforward physics model simulates multi-contact experiments that are too costly to acquire in reality and a practical automatic indentation setup acquires real but geometrically limited multi-contact data. Although the number of real measurements required to learn the discrepancy between the sensor and the model is still large due to the inherent inverse nature of ERT-based tactile sensors, our combination of simulation and deep networks achieves better performance than either physical modeling or learning alone. This approach can advance practical large-area tactile sensing for industrial automation systems where multiple contacts occur, such as in manufacturing and assistive robotics. It could also likely be adapted to other nonlinear inverse problems.

Hugs are complex affective interactions that often include gestures like squeezes. We present six new guidelines for designing interactive hugging robots, which we validate through two studies with our custom robot. To achieve autonomy, we investigated robot responses to four human intra-hug gestures: holding, rubbing, patting, and squeezing. A Total of 32 users each exchanged and rated 16 hugs with an experimenter-controlled HuggieBot 2.0. The robot’s inflated torso’s microphone and pressure sensor collected data of the subjects’ demonstrations that were used to develop a perceptual algorithm that classifies user actions with 88% accuracy. Users enjoyed robot squeezes, regardless of their performed action, they valued variety in the robot response, and they appreciated robot-initiated intra-hug gestures. From average user ratings, we created a probabilistic behavior algorithm that chooses robot responses in real time. We implemented improvements to the robot platform to create HuggieBot 3.0 and then validated its gesture perception system and behavior algorithm with 16 users. The robot’s responses and proactive gestures were greatly enjoyed. Users found the robot more natural, enjoyable, and intelligent in the last phase of the experiment than in the first. After the study, they felt more understood by the robot and thought robots were nicer to hug.

Screen-based pedestrian navigation assistance can be distracting or inaccessible to users. Shape-changing haptic interfaces can overcome these concerns. The S-BAN is a new handheld haptic interface that utilizes a parallel kinematic structure to deliver 2-DOF spatial information over a continuous workspace, with a form factor suited to integration with other travel aids. The ability to pivot, extend and retract its body opens possibilities and questions around spatial data representation. We present a static study to understand user perception of absolute pose and relative motion for two spatial mappings, showing the highest sensitivity to relative motions in the cardinal directions. We then present an embodied navigation experiment in virtual reality (VR). User motion efficiency when guided by the S-BAN was statistically equivalent to using a vision-based tool (a smartphone proxy). Although haptic trials were slower than visual trials, participants’ heads were more elevated with the S-BAN, allowing greater visual focus on the environment.

Grounded force-feedback (GFF) devices are an established and diverse class of haptic technology based on robotic arms. However, the number of designs and how they are specified make comparing devices difficult. We thus present Haptify, a benchmarking system that can thoroughly, fairly, and noninvasively evaluate GFF haptic devices. The user holds the instrumented device end-effector and moves it through a series of passive and active experiments. Haptify records the interaction between the hand, device, and ground with a seven-camera optical motion-capture system, a 60-cm-square custom force plate, and a customized sensing end-effector. We demonstrate six key ways to assess GFF device performance: workspace shape, global free-space forces, global free-space vibrations, local dynamic forces and torques, frictionless surface rendering, and stiffness rendering. We then use Haptify to benchmark two commercial haptic devices. With a smaller workspace than the 3D Systems Touch, the more expensive Touch X outputs smaller free-space forces and vibrations, smaller and more predictable dynamic forces and torques, and higher-quality renderings of a frictionless surface and high stiffness.

A human handing over an object modulates their grasp and movements to accommodate their partner's capa-bilities, which greatly increases the likelihood of a successful transfer. State-of-the-art robot behavior lacks this level of user understanding, resulting in interactions that force the human partner to shoulder the burden of adaptation. This paper investigates how visual occlusion of the object being passed affects the subjective perception and quantitative performance of the human receiver. We performed an experiment in virtual reality where seventeen participants were tasked with repeatedly reaching to take a tool from the hand of a robot; each of the three tested objects (hammer, screwdriver, scissors) was presented in a wide variety of poses. We carefully analysed the user's hand and head motions, the time to grasp the object, and the chosen grasp location, as well as participants' ratings of the grasp they just performed. Results show that initial visibility of the handle significantly increases the reported holdability and immediate usability of a tool. Furthermore, a robot that offers objects so that their handles are more occluded forces the receiver to spend more time in planning and executing the grasp and also lowers the probability that the tool will be grasped by the handle. Together these findings indicate that robots can more effectively support their human work partners by increasing the visibility of the intended grasp location of objects being passed.

Knee osteoarthritis is a progressive disease mediated by high joint loads. Foot progression angle modifications that reduce the knee adduction moment (KAM), a surrogate of knee loading, have demonstrated efficacy in alleviating pain and improving function. Although changes to the foot progression angle are overall beneficial, KAM reductions are not consistent across patients. Moreover, customized interventions are time-consuming and require instrumentation not commonly available in the clinic. We present a regression model that uses minimal clinical data-a set of six features easily obtained in the clinic-to predict the extent of first peak KAM reduction after toe-in gait retraining. For such a model to generalize, the training data must be large and variable. Given the lack of large public datasets that contain different gaits for the same patient, we generated this dataset synthetically. Insights learned from a ground-truth dataset with both baseline and toe-in gait trials (N = 12) enabled the creation of a large (N = 138) synthetic dataset for training the predictive model. On a test set of data collected by a separate research group (N = 15), the first peak KAM reduction was predicted with a mean absolute error of 0.134% body weight * height (%BW*HT). This error is smaller than the standard deviation of the first peak KAM during baseline walking averaged across test subjects (0.306%BW*HT). This work demonstrates the feasibility of training predictive models with synthetic data and provides clinicians with a new tool to predict the outcome of patient-specific gait retraining without requiring gait lab instrumentation.

Compared to grounded force feedback, providing tactile feedback via a wearable device can free the user and broaden the potential applications of simulated physical interactions. However, neither the limitations nor the full potential of tactile-only feedback have been precisely examined. Here we investigate how the dimensionality of cutaneous fingertip feedback affects user movements and virtual object recognition. We combine a recently invented 6-DOF fingertip device with motion tracking, a head-mounted display, and novel contact-rendering algorithms to enable a user to tactilely explore immersive virtual environments. We evaluate rudimentary 1-DOF, moderate 3-DOF, and complex 6-DOF tactile feedback during shape discrimination and mass discrimination, also comparing to interactions with real objects. Results from 20 naive study participants show that higher-dimensional tactile feedback may indeed allow completion of a wider range of virtual tasks, but that feedback dimensionality surprisingly does not greatly affect the exploratory techniques employed by the user.

Receiving a hug is one of the best ways to feel socially supported, and the lack of social touch can have severe negative effects on an individual's well-being. Based on previous research both within and outside of HRI, we propose six tenets ("commandments") of natural and enjoyable robotic hugging: a hugging robot should be soft, be warm, be human sized, visually perceive its user, adjust its embrace to the user's size and position, and reliably release when the user wants to end the hug. Prior work validated the first two tenets, and the final four are new. We followed all six tenets to create a new robotic platform, HuggieBot 2.0, that has a soft, warm, inflated body (HuggieChest) and uses visual and haptic sensing to deliver closed-loop hugging. We first verified the outward appeal of this platform in comparison to the previous PR2-based HuggieBot 1.0 via an online video-watching study involving 117 users. We then conducted an in-person experiment in which 32 users each exchanged eight hugs with HuggieBot 2.0, experiencing all combinations of visual hug initiation, haptic sizing, and haptic releasing. The results show that adding haptic reactivity definitively improves user perception a hugging robot, largely verifying our four new tenets and illuminating several interesting opportunities for further improvement.

The synthesis of realistic robot grasps in a simulated environment is pivotal in generating datasets that support sim-to-real transfer learning. In a step toward achieving this goal, we propose PrendoSim, an open-source grasp generator based on a proxy-hand simulation that employs NVIDIA’s physics engine (PhysX) and the recently released articulated-body objects developed by Unity (https://prendosim.github.io). We present the implementation details, the method used to generate grasps, the approach to operationally evaluate stability of the generated grasps, and examples of grasps obtained with two different grippers (a parallel jaw gripper and a three-finger hand) grasping three objects selected from the YCB dataset (hammer, screwdriver, and scissors). Compared to simulators proposed in the literature, PrendoSim balances grasp realism and ease of use, displaying an intuitive interface and enabling the user to produce a large and varied dataset of stable grasps.

Robots hold great potential for supporting exercise and physical therapy, but such systems are often cumbersome to set up and require expert supervision. We aim to solve these concerns by combining Captury Live, a real-time markerless motion-capture system, with a Rethink Robotics Baxter Research Robot to create the Robot Interaction Studio. We evaluated this platform for unsupervised human-robot interaction (HRI) through a 75-minute-long user study with seven adults who were given minimal instructions and no feedback about their actions. The robot used sounds, facial expressions, facial colors, head motions, and arm motions to sequentially present three categories of cues in randomized order while constantly rotating its face screen to look at the user. Analysis of the captured user motions shows that the cue type significantly affected the distance subjects traveled and the amount of time they spent within the robot’s reachable workspace, in alignment with the design of the cues. Heat map visualizations of the recorded user hand positions confirm that users tended to mimic the robot’s arm poses. Despite some initial frustration, taking part in this study did not significantly change user opinions of the robot. We reflect on the advantages of the proposed approach to unsupervised HRI as well as the limitations and possible future extensions of our system.

The lack of haptically aware upper-limb prostheses forces amputees to rely largely on visual cues to complete activities of daily living. In contrast, non-amputees inherently rely on conscious haptic perception and automatic tactile reflexes to govern volitional actions in situations that do not allow for constant visual attention. We therefore propose a myoelectric prosthesis system that reflects these concepts to aid manipulation performance without direct vision. To implement this design, we constructed two fabric-based tactile sensors that measure contact location along the palmar and dorsal sides of the prosthetic fingers and grasp pressure at the tip of the prosthetic thumb. Inspired by the natural sensorimotor system, we use the measurements from these sensors to provide vibrotactile feedback of contact location and implement a tactile grasp controller with reflexes that prevent over-grasping and object slip. We compare this tactile system to a standard myoelectric prosthesis in a challenging reach-to-pick-and-place task conducted without direct vision; 17 non-amputee adults took part in this single-session between-subjects study. Participants in the tactile group achieved more consistent high performance compared to participants in the standard group. These results show that adding contact-location feedback and reflex control increases the consistency with which objects can be grasped and moved without direct vision in upper-limb prosthetics.

Humans rely on categories to mentally organize and understand sets of complex objects. One such set, haptic devices, has myriad technical attributes that affect user experience in complex ways. Seeking an effective navigation structure for a large online collection, we elicited expert mental categories for grounded force-feedback haptic devices: 18 experts (9 device creators, 9 interaction designers) reviewed, grouped, and described 75 devices according to their similarity in a custom card-sorting study. From the resulting quantitative and qualitative data, we identify prominent patterns of tagging versus binning, and we report 6 uber-attributes that the experts used to group the devices, favoring affordances over device specifications. Finally, we derive 7 device categories and 9 subcategories that reflect the imperfect yet semantic nature of the expert mental models. We visualize these device categories and similarities in the online haptic collection, and we offer insights for studying expert understanding of other human-centered technology.

Tactile sensors based on electrical resistance tomography (ERT) have shown many advantages for implementing a soft and scalable whole-body robotic skin; however, calibration is challenging because pressure reconstruction is an ill-posed inverse problem. This paper introduces a method for calibrating soft ERT-based tactile sensors using sim-to-real transfer learning with a finite element multiphysics model. The model is composed of three simple models that together map contact pressure distributions to voltage measurements. We optimized the model parameters to reduce the gap between the simulation and reality. As a preliminary study, we discretized the sensing points into a 6 by 6 grid and synthesized single- and two-point contact datasets from the multiphysics model. We obtained another single-point dataset using the real sensor with the same contact location and force used in the simulation. Our new deep neural network architecture uses a de-noising network to capture the simulation-to-real gap and a reconstruction network to estimate contact force from voltage measurements. The proposed approach showed 82% hit rate for localization and 0.51 N of force estimation error performance in singlecontact tests and 78.5% hit rate for localization and 5.0 N of force estimation error in two-point contact tests. We believe this new calibration method has the possibility to improve the sensing performance of ERT-based tactile sensors.

Creating haptic experiences often entails inventing, modifying, or selecting specialized hardware. However, interaction designers are rarely engineers, and 30 years of haptic inventions are buried in a fragmented literature that describes devices mechanically rather than by potential purpose. We conceived of Haptipedia to unlock this trove of examples: Haptipedia presents a device corpus for exploration through metadata that matter to both device and interaction designers. It is a taxonomy of device attributes that go beyond physical description to capture potential utility, applied to a growing database of 105 grounded force-feedback devices, and accessed through a public visualization that links utility to morphology. Haptipedia's design was driven by both systematic review of the haptic device literature and rich input from diverse haptic designers. We describe Haptipedia's reception (including hopes it will redefine device reporting standards) and our plans for its sustainability through community participation.

We live in an aging society; social-physical human-robot interaction has the potential to keep our older adults healthy by motivating them to exercise. After summarizing prior work, this paper proposes a tool that can be used to design exercise and therapy interactions to be performed by an upper-body humanoid robot. The interaction design tool comprises a teleoperation system that transmits the operator's arm motions, head motions and facial expression along with an interface to monitor and assess the motion of the user interacting with the robot. We plan to use this platform to create dynamic and intuitive exercise interactions.

Robots operating in unstructured environments would benefit from soft whole-body tactile sensors, but implementing such systems typically requires complex electrical wiring to a large number of sensing elements. The reconstruction method called electrical resistance tomography (ERT) has shown promising results (good coverage, manufacturability, and robustness) using electrodes located only along the boundary of the sensing region. However, relatively poor spatial resolution in the sensor's central region is a major drawback of the ERT approach. This paper introduces a new scheme of internal array electrodes to improve spatial resolution. We also systematically derive the optimal pairwise current injection patterns from a mathematical formulation of the ERT system. By highlighting the importance of each electrode pair, this approach enabled us to reduce the number of current injection patterns. Simulation of the standard and proposed sensor designs revealed that the internal array electrodes greatly improve distinguishability in the central region. For validation, a fabric-based soft tactile sensor made of multiple conductive fabrics was developed, including electronics that enable sampling at 200 Hz. During a 225-point localization test conducted without sensor-specific calibration, the constructed sensor showed average localization errors of 2.85 cm ± 1.02 cm. This result is notable because only 16 point electrodes were used to achieve this performance.

Humans can form an impression of how a new object feels simply by touching its surfaces with the densely innervated skin of the fingertips. Many haptics researchers have recently been working to endow robots with similar levels of haptic intelligence, but these efforts almost always employ hand-crafted features, which are brittle, and concrete tasks, such as object recognition. We applied unsupervised feature learning methods, specifically K-SVD and Spatio-Temporal Hierarchical Matching Pursuit (ST-HMP), to rich multi-modal haptic data from a diverse dataset. We then tested the learned features on 19 more abstract binary classification tasks that center on haptic adjectives such as smooth and squishy. The learned features proved superior to traditional hand-crafted features by a large margin, almost doubling the average F1score across all adjectives. Additionally, particular exploratory procedures (EPs) and sensor channels were found to support perception of certain haptic adjectives, underlining the need for diverse interactions and multi-modal haptic data.

Touching an object causes rich haptic cues that enable you to understand the object's physical properties and adeptly control the interaction. Although human experience centers on physical contact with tangible items, few computer systems provide the user with high-fidelity touch feedback, limiting their intuitiveness. Haptic interfaces are mechatronic systems that modulate the physical interaction between a human and his or her tangible surroundings. Such interfaces typically involve mechanical, electrical, and computational layers that work together to sense user motions or forces, quickly process these inputs with other information, and physically respond by actuating elements of the user's surroundings. By way of three examples, this talk will demonstrate that well-designed tactile feedback can greatly increase the realism of virtual worlds. First, we created a simple visuo-audio-tactile simulator to help dental students learn to find cavities in teeth. The user watches a video of a real dental tool interacting with a tooth while simultaneously feeling an authentic rendering of the associated contact vibrations. Second, we created the world's most realistic haptic virtual surfaces by recording and modeling what a user feels when touching 100 real objects with an instrumented stylus. The perceptual effects of displaying the resulting data-driven friction forces, tapping transients, and texture vibrations were quantified by having users compare the original surfaces to their virtual versions. Third, we extended the haptic texture concept to capture how a real robot vibrates as it moves its joints and tied this model to measured user motions. The resulting vibrotactile experiences were formally evaluated and then added to an immersive game that lets the user feel what it would be like to turn into a robot. While much work remains to be done, we are starting to see the tantalizing potential of systems that leverage tactile cues to allow a user to interact with virtual environments as though they were real.

The Portable Robotic Optical/Tactile ObservatioN PACKage (PROTONPACK, or Proton for short) is a new handheld visuo-haptic sensing system that records surface interactions. We previously demonstrated system calibration and a classification task using external motion tracking. This paper details improvements in surface classification performance and removal of the dependence on external motion tracking, necessary before embarking on our goal of gathering a vast surface interaction dataset. Two experiments were performed to refine data collection parameters. After adjusting the placement and filtering of the Proton's high-bandwidth accelerometers, we recorded interactions between two differently-sized steel tooling ball end-effectors (diameter 6.35 and 9.525 mm) and five surfaces. Using features based on normal force, tangential force, end-effector speed, and contact vibration, we trained multi-class SVMs to classify the surfaces using 50 ms chunks of data from each end-effector. Classification accuracies of 84.5% and 91.5% respectively were achieved on unseen test data, an improvement over prior results. In parallel, we pursued on-board motion tracking, using the Proton's camera and fiducial markers. Motion tracks from the external and onboard trackers agree within 2 mm and 0.01 rad RMS, and the accuracy decreases only slightly to 87.7% when using onboard tracking for the 9.525 mm end-effector. These experiments indicate that the Proton 2 is ready for portable data collection.

Robotic-assisted surgery requires an intuitive and effective human-machine interface. Providing haptic feedback for pinching and dissecting motions of bipolar forceps, a tool commonly used in neurosurgery, could potentially improve the surgeon's experience. Current haptic hand controllers have limited actuation and feedback capability, requiring surgeons to hold the handle differently compared to a conventional tool. This paper presents a new master design that provides 1-DOF force feedback by adding a Hall-effect sensor and a voice coil actuator directly onto a bipolar forceps. Twenty participants used this interface to perform a remote stiffness perception test that employed the method of constant stimuli. Ten participants pinched the samples, and the other ten dissected them. Each participant did two blocks of 35 trials with only visual feedback or with visual and haptic feedback in random order. Psychometric functions were created from the results to compare perceptual capabilities, metrics were calculated from the force and position data, and participant survey responses were analyzed. The results show that providing the force feedback made the task seem easier, increased the participant's confidence, and reduced the total tip distance traveled in the pinching task. The haptic feedback slightly improved stiffness perception in the pinching task but did not improve perception in the dissection task. These results support the utility of a force-feedback attachment to conventional forceps for pinching and motivate further investigation into the design for dissection.

Robots which interact with the physical world will benefit from a fine-grained tactile understanding of objects and surfaces. Additionally, for certain tasks, robots may need to know the haptic properties of an object before touching it. To enable better tactile understanding for robots, we propose a method of classifying surfaces with haptic adjectives (e.g., compressible or smooth) from both visual and physical interaction data. Humans typically combine visual predictions and feedback from physical interactions to accurately predict haptic properties and interact with the world. Inspired by this cognitive pattern, we propose and explore a purely visual haptic prediction model. Purely visual models enable a robot to “feel” without physical interaction. Furthermore, we demonstrate that using both visual and physical interaction signals together yields more accurate haptic classification. Our models take advantage of recent advances in deep neural networks by employing a unified approach to learning features for physical interaction and visual observations. Even though we employ little domain specific knowledge, our model still achieves better results than methods based on hand-designed features.

All over the world, people find joy and amusement in playing hand-clapping games such as “Pat-a-cake” and “Slide.” Thus, as robots enter everyday human spaces and work together with people, we see potential for them to entertain, engage, and assist humans through cooperative clapping games. This paper explores how data recorded from a pair of commonly available inertial measurement units (IMUs) worn on a human's hands can contribute to the teaching of a hand-clapping robot. We identified representative hand-clapping activities, considered approaches to classify games, and conducted a study to record hand-clapping motion data. Analysis of data from fifteen participants indicates that support vector machines and Markov chain analysis can correctly classify 95.5% of the demonstrated hand-clapping motions (from ten discrete actions) and 92.3% of the hand-clapping game demonstrations recorded in the study. These results were calculated by withholding a participant's entire dataset for testing, so these results should represent general system behavior for new users. Overall, this research lays the groundwork for a simple and efficient method that people could use to demonstrate hand-clapping games to robots.

Human friends and teammates commonly connect through handshakes, high fives, fist bumps, and other forms of hand-to-hand contact. As robots enter everyday human spaces, they will have the opportunity to join in such physical interactions, but few current robots are intended to touch humans. To begin investigating this topic, we sought to discover precisely how robots should move and react in hand-clapping games, which we define as interactions involving repeated hand-to-hand contacts between two agents. We conducted an experiment to observe seven pairs of people performing a variety of hand-clapping activities. Their recorded hand movements were accurately described by sinusoids that have a constant participant-specific maximum velocity across clapping tempos. Behaviorally, people struggled most with hand clapping at fast tempos, but they also smiled and laughed most often during fast trials. We used the human-human experiment findings to select, modify, and program a Rethink Robotics Baxter Research Robot to clap hands with a human partner. Preliminary tests have demonstrated that this robot can move like our participants and reliably detect human hand impacts through its wrist-mounted accelerometers, thereby exhibiting promise as a safe and engaging interaction partner.

Dragging a tool across a textured surface produces vibrations that convey important perceptual information about the interaction and the underlying qualities of the surface. These vibrations depend on the motions of the tool and respond to both normal force and tangential speed. This paper explores various methods of simulating haptic texture interactions by rendering tool vibrations that are based on recorded data. We designed and ran a human-subject study (N=15) to analyze the importance of creating virtual texture vibrations that respond to user force and speed. Our analysis of data from fifteen textures showed that removing speed responsiveness did cause a statistically significant decrease in perceived realism, but removing force responsiveness did not. This result indicates that virtual textures aiming to simulate real surfaces should vary the rendered vibrations with user speed but may not need to vary them with user force.

Creating a robot that can teach humans simple interactive tasks such as high-fiving requires research at the intersection of physical human-robot interaction (PHRI) and socially assistive robotics. This paper shows how observation of natural human-human interaction can improve the design of requirements for social-physical robots and form a framework for autonomous execution of interactive physical tasks. Eleven pairs of human subjects were recruited to perform a set of high-fiving games; a magnetic motion tracker and an accelerometer were mounted to each person's hand for the duration of the experiment, and each subject completed several questionnaires about the experience. The results reveal valuable clues about the generally positive feelings of the participants and the movement of their hands during play. We discuss how we plan to use these results to create a robot that can teach humans similar high-fiving games.

To moderate oral presentations a chair must manage time, and communicate time parameters to speakers through a variety of means. But speakers often miss time cues, chairs cannot confirm their receipt, and the broken dialogue can be a sideshow for the audience. We developed HaNS, a wireless wrist-worn chair-speaker Haptic Notification System that delivers tactile cues for time-managing oral presentations, and performed field observations at university research seminars and two mid-sized academic conferences (input from 66 speakers, 21 chairs, and 65 audience members). Results indicate that HaNS can improve a user's awareness of time, facilitate chair-speaker coordination, and reduce distraction of speaker and audience through its private communication channel. Eliminating overruns will require improvement in speaker 'internal' control, which our results suggest HaNS can also support given practice. We conclude with design guidelines for both conference-deployed and personal timing tools, using touch or another notification modality.

If you pick up a tool and drag its tip across a table, a rock, or a swatch of fabric, you are able to feel variations in the textures even though you are not directly touching them. These vibrations are characteristic of the material and the motions made when interacting with the surface. This paper presents a new method for creating haptic texture models from data recorded during natural and unconstrained motions using a new haptic recording device. The recorded vibration data is parsed into short segments that represent the feel of the surface at the associated tool force and speed. We create a low-order auto-regressive (AR) model for each data segment and construct a Delaunay triangulation of models in force-speed space for each surface. During texture rendering, we stably interpolate between these models using barycentric coordinates and drive the interpolated model with white noise to output synthetic vibrations. Our methods were validated through application to data recorded by eight human subjects and the experimenter interacting with six textures. We present a new spectral metric for determining perceptual match of the models in order to evaluate the effectiveness and consistency of the segmenting and modeling approach. Multidimensional scaling (MDS) on the pairwise differences in the synthesized vibrations shows that the 54 created texture models cluster by texture in a two-dimensional perceptual space.

Humans can feel fine surface details through a rigid probe by attending to the high-frequency vibrations caused by contact. To aid our efforts in creating realistic virtual textures, we sought to understand the dynamics of such interactions through a combination of data collection and offline simulation. A handheld probe was fitted with interchangeable hemispherical tool tips, accelerometers, a force sensor, and a magnetic motion tracker and then used to record interactions with a sinusoidal grating. We mathematically modeled this system using planar rigid-body dynamics, a lumped-parameter model of the human hand, an offset surface for collision detection, and Hertzian contact mechanics at the point(s) of interaction. Tuning the parameters of the hand enabled the simulation output to closely match the lateral and axial tool accelerations recorded at four combinations of tool tip size and tool speed.

Delivering on the promise of real-world robotics will require robots that can communicate with humans through natural language by learning new words and concepts through their daily experiences. Our research strives to create a robot that can learn the meaning of haptic adjectives by directly touching objects. By equipping the PR2 humanoid robot with state-of-the-art biomimetic tactile sensors that measure temperature, pressure, and fingertip deformations, we created a platform uniquely capable of feeling the physical properties of everyday objects. The robot used five exploratory procedures to touch 51 objects that were annotated by human participants with 34 binary adjective labels. We present both static and dynamic learning methods to discover the meaning of these adjectives from the labeled objects, achieving average F1 scores of 0.57 and 0.79 on a set of eight previously unfelt items.

This paper reviews the technology and applications of vibrotactile display, an effective information transfer modality for the emerging area of haptic media. Our emphasis is on summarizing foundational knowledge in this area and providing implementation guidelines for application designers who do not yet have a background in haptics. Specifically, we explain the relevant human vibrotactile perceptual capabilities, detail the main types of commercial vibrotactile actuators, and describe how to build both monolithic and localized vibrotactile displays. We then identify exemplary vibrotactile display systems in application areas ranging from the presentation of physical object properties to broadcasting vibrotactile media content.

Manipulating objects with an upper limb prosthesis requires significantly more visual attention than doing the same task with an intact limb. Prior work and comments from individuals lacking proprioception indicate that conveying prosthesis motion through a nonvisual sensory channel would reduce and possibly remove the need to watch the prosthesis. To motivate the design of suitable sensory substitution devices, this study investigates the difference between seeing a virtual prosthetic limb move and feeling one's real limb move. Fifteen intact subjects controlled a virtual prosthetic finger in a one-degree-of-freedom rotational spring discrimination task. A custom haptic device was used to measure both real finger position and applied finger force, and the resulting prosthetic finger movement was displayed visually (on a computer screen) and/or proprioceptively (by allowing the subject's real finger to move). Spring discrimination performance was tested for three experimental sensory conditions-visual motion, proprioceptive motion, and visual and proprioceptive motion-using the method of constant stimuli, with a reference stiffness of 290 N/m. During each trial, subjects sequentially pressed the right index finger on a pair of hard-surfaced virtual springs and decided which was stiffer. No significant performance differences were found between the three experimental sensory conditions, but subjects perceived proprioceptive motion to be significantly more useful than visual motion. These results imply that relaying proprioceptive information through a nonvisual channel could reduce visual attention during prosthesis control while maintaining task performance, thus improving the upper limb prosthesis experience.

Vibrotactile signals convey the touch-based characteristics of object surfaces felt through a tool. They particularly enhance the quality of human-machine interactions by providing realistic haptic perception of textures. In this paper, inspired by the similarities observed between vibrotactile texture signals and speech signals, we present a novel vibrotactile texture codec for bilateral teleoperation, based on well-known speech coding techniques. The proposed low bitrate, high quality codec preserves not only the spectral signature vital to the general feel of the texture, but also important temporal features of the texture signal. We report a compression ratio of 8:1 (12.5 %) with a constant output bitrate of 4 kbps, and we validate the perceptual transparency of the codec via rigorous subjective tests and analyses.

We present and evaluate an arm-motion guidance system that uses magnetic tracking sensors and low cost vibrotactile actuators. The system measures the movement of the user's arm and provides vibration feedback at the wrist and elbow when they stray from the desired motion. An initial study was conducted to investigate whether adding tactile feedback to visual feedback reduces motion errors when a user is learning a new arm trajectory. Although subjects preferred it, we found that the addition of tactile feedback did not affect motion tracking performance. We also found no strong preference or performance differences between attractive and repulsive tactile feedback. Some factors that may have influenced these results include the speed and the complexity of the tested motions, the type of tactile actuators and drive signals used, and inconsistencies in joint angle estimation due to Euler angle gimbal lock. We discuss insights from this analysis and provide suggestions for future systems and studies in tactile motion guidance.

Individuals learning a new sport often repeat a motion hundreds or thousands of times to try to perfect their form. The quintessential example of this process may be a beginning golfer struggling to learn to putt, where strokes must be precise and consistent in order to place the ball in the hole. This paper presents a four-cable haptic device designed to help golfers learn to improve their putting accuracy. This planar three-DOF system provides feedback that consists of two Cartesian forces and one angular moment. We present the system's design and kinematics, along with a closed-loop controller that helps the user keep the putter head at the correct angle in the plane. We evaluated our design through a study in which five subjects used the system to repeatedly putt at a target both with and without assistance. While assistance did not change the mean of the putting distribution, it did significantly affect the variance for some subjects.

We present four wearable tactile actuators capable of recreating physical sensations commonly experienced in human interactions, including tapping on, dragging across, squeezing, and twisting an individual's wrist. In seeking to create tactile signals that feel natural and are easy to understand, we developed movement control interfaces to play back each of these forms of actual human physical contact. Through iterative design, prototyping, programming, and testing, each of these servo-motor-based mechanisms produces a signal that is gradable in magnitude, can be played in a variety of temporal patterns, is localizable to a small area of skin, and, for three of the four actuators, has an associated direction. Additionally, we have tried to design toward many of the characteristics that have made high frequency vibration the most common form of wearable tactile feedback, including low cost, light weight, comfort, and small size. Bolstered by largely positive comments from naive users during an informal testing session, we plan to continue improving these devices for future use in tactile motion guidance.

We present a novel robotic grasp controller that allows a sensorized parallel jaw gripper to gently pick up and set down unknown objects once a grasp location has been selected. Our approach is inspired by the control scheme that humans employ for such actions, which is known to centrally depend on tactile sensation rather than vision or proprioception. Our controller processes measurements from the gripper's fingertip pressure arrays and hand-mounted accelerometer in real time to generate robotic tactile signals that are designed to mimic human SA-I, FA-I, and FA-II channels. These signals are combined into tactile event cues that drive the transitions between six discrete states in the grasp controller: Close, Load, Lift and Hold, Replace, Unload, and Open. The controller selects an appropriate initial grasping force, detects when an object is slipping from the grasp, increases the grasp force as needed, and judges when to release an object to set it down. We demonstrate the promise of our approach through implementation on the PR2 robotic platform, including grasp testing on a large number of real-world objects.

Sliding a probe over a textured surface generates a rich collection of vibrations that one can easily use to create a mental model of the surface. Haptic virtual environments attempt to mimic these real interactions, but common haptic rendering techniques typically fail to reproduce the sensations that are encountered during texture exploration. Past approaches have focused on building a representation of textures using a priori ideas about surface properties. Instead, this paper describes a process of synthesizing probe-surface interactions from data recorded from real interactions. We explain how to apply the mathematical principles of Linear Predictive Coding (LPC) to develop a discrete transfer function that represents the acceleration response under specific probe-surface interaction conditions. We then use this predictive transfer function to generate unique acceleration signals of arbitrary length. In order to move between transfer functions from different probe-surface interaction conditions, we develop a method for interpolating the variables involved in the texture synthesis process. Finally, we compare the results of this process with real recorded acceleration signals, and we show that the two correlate strongly in the frequency domain.

This paper outlines how a control moment gyroscope can be used to generate haptic torque feedback while minimizing the effects of a constrained gimbal workspace. We present the design of the iTorqU 2.1 and discuss how it compares to previously developed systems. We then detail the control algorithms we have developed for both transparency and torque output modes. The prescribed transparency controller is typical in design, but the torque output algorithm is novel to this type of haptic device. It makes use of a series of position-p-at-time-t commands, which we call packets. Five packet designs were considered in this research, but we have included only the most important three in this paper. While this research deals with torque feedback, it ultimately presents a method for working with devices that are limited by the need for continuous reset to a home position before subsequent outputs.

Proprioception plays a crucial role in enabling humans to move purposively and interact with their physical surroundings. Current technology in upper-limb prostheses, while beginning to incorporate some haptic feedback, does not provide amputees with proprioceptive information about the state of the limb. Thus, the wearer must visually monitor the limb, which is often inconvenient or even impossible for some tasks. This work seeks to quantify the potential benefits of incorporating proprioceptive motion feedback into upper-limb prosthesis designs. We apply a noninvasive method for controlling the availability of proprioceptive motion feedback in unimpaired individuals in a human subject study to compare the benefits of visual and proprioceptive motion feedback in targeted motion tasks. Combined results of the current study and our previous study using a different task indicate that the addition of proprioceptive motion feedback improves targeting accuracy under nonsighted conditions and, for some tasks, under sighted conditions as well. This work motivates the development of methods for providing artificial proprioceptive feedback to a prosthesis wearer.

Almost all commercially available haptic interfaces share a common pitfall, a small shoebox-sized workspace; these devices typically rely on rigid-link manipulator design concepts. In this paper we outline our design for a new kinesthetic haptic system that drastically increases the usable haptic workspace. We present a proof-of-concept prototype, along with our analysis of its capabilities. Our design uses optical tracking to sense the position of the device, and air jet actuation to generate forces. By combining these two technologies, we are able to detach our device from the ground, thus sidestepping many problems that have plagued traditional haptic devices including workspace size, friction, and inertia. We show that optical tracking and air jet actuation successfully enable kinesthetic haptic interaction with virtual environments. Given an appropriately large volume high-pressure air source, and a reasonably high speed tracking system, this design paradigm has many desirable qualities when compared to traditional haptic design schemes.

High frequency contact accelerations convey important information that the vast majority of haptic interfaces cannot render. Building on prior work, we present an approach to haptic interface design that uses a dedicated linear voice coil actuator and a dynamic system model to allow the user to feel these signals. This approach was tested through use in a bilateral teleoperation experiment where a user explored three textured surfaces under three different acceleration control architectures: none, constant gain, and dynamic compensation. The controllers that use the dedicated actuator vastly outperform traditional position-position control at conveying realistic contact accelerations. Analysis of root mean square error, linear regression, and discrete Fourier transforms of the acceleration data also indicate a slight performance benefit for dynamic compensation over constant gain.

Humans rely on information-laden high-frequency accelerations in addition to quasi-static forces when interacting with objects via a handheld tool. Telerobotic systems have traditionally struggled to portray such contact transients due to closed-loop bandwidth and stability limitations, leaving remote objects feeling soft and undefined. This work seeks to maximize the user's feel for the environment through the approach of acceleration matching; high-frequency fingertip accelerations are combined with standard low-frequency position feedback without requiring a secondary actuator on the master device. In this method, the natural dynamics of the master are identified offline using frequency-domain techniques, estimating the relationship between commanded motor current and handle acceleration while a user holds the device. During subsequent telerobotic interactions, a high-bandwidth sensor measures accelerations at the slave's end effector, and the real-time controller re-creates these important signals at the master handle by inverting the identified model. The details of this approach are explored herein, and its ability to render hard and rough surfaces is demonstrated on a standard master-slave system. Combining high-frequency acceleration matching with position-error-based feedback of quasi-static forces creates a hybrid signal that closely corresponds to human sensing capabilities, instilling telerobotics with a more realistic sense of remote touch

Tapping on surfaces in a typical virtual environment feels like contact with soft foam rather than a hard object. The realism of such interactions can be dramatically improved by superimposing event-based, high-frequency transient forces over traditional position-based feedback. When scaled by impact velocity, hand-tuned pulses and decaying sinusoids produce haptic cues that resemble those experienced during real impacts. Our new method for generating appropriate transients inverts a dynamic model of the haptic device to determine the motor forces required to create prerecorded acceleration profiles at the user's fingertips. After development, the event-based haptic paradigm and the method of acceleration matching were evaluated in a carefully controlled user study. Sixteen individuals blindly tapped on nine virtual and three real samples, rating the degree to which each felt like real wood. Event-based feedback achieved significantly higher realism ratings than the traditional rendering method. The display of transient signals made virtual objects feel similar to a real sample of wood on a foam substrate, while position feedback alone received ratings similar to those of foam. This work provides an important new avenue for increasing the realism of contact in haptic interactions.

Providing the user with high-fidelity force feedback has persistently challenged the field of telerobotics. Interaction forces measured at the remote site and displayed to the user cause unintended master device motion. This movement is interpreted as a command for the slave robot and can drive the closed-loop system unstable. This paper builds on a recently proposed approach for achieving stable, high-gain force reflection via cancellation of the master mech anism’s induced motion. Such a strategy hinges on obtaining a good model of the master’s response to force feedback. Herein, we present a thorough modeling approach based on successive isolation of system components, demonstrated on a one-degree-of-freedom testbed. A sixth-order mechanical model, including viscous and Coulomb friction as well as a new method for modeling hysteretic stiffness, describes the testbed’s high-frequency resonant modes. This modeling method’s ability to predict induced master motion should lead to significant improvements in force-reflecting teleoperation via the cancellation approach.