For decades, robotics researchers have pursued various tasks for multi-robot systems, from cooperative manipulation to search and rescue. These tasks are multi-robot extensions of classical robotic tasks and often optimized on dimensions such as speed or efficiency. As robots transition from commercial and research settings into everyday environments, social task aims such as engagement or entertainment become increasingly relevant. This work presents a designerly contribution—building a multi-robot task in which the main aim is to enthrall and interest. In this task, the goal is for a human to be drawn to move alongside and participate in a dynamic, expressive robot flock. Towards this aim, the research team created algorithms for robot movements and engaging interaction modes such as gestures and sound. The contributions are as follows: (1) a novel group navigation algorithm involving human and robot agents, (2) a gesture responsive algorithm for real-time, human–robot flocking interaction, (3) a weight mode characterization system for modifying flocking behavior, and (4) a method of encoding a choreographer’s preferences inside a dynamic, adaptive, learned system. An experiment was performed to understand individual human behavior while interacting with the flock under three conditions: weight modes selected by a human choreographer, a learned model, or subset list. Results from the experiment indicated that the perception of the experience was not influenced by the weight mode selection. This work elucidates how differing task aims such as engagement manifest in multi-robot system design and execution, and broadens the domain of multi-robot tasks.

Human crowd simulation in virtual reality (VR) is a powerful tool with potential applications including emergency evacuation training and assessment of building layout. While haptic feedback in VR enhances immersive experience, its effect on virtual walking behavior in dense and dynamic pedestrian flows is unknown. Through a user study, we investigated how haptic feedback changes user walking motion in crowded pedestrian flows in VR. The results indicate that haptic feedback changed users’ collision avoidance movements, as measured by increased walking trajectory length and change in pelvis angle. The displacements of users’ lateral position and pelvis angle were also increased in the instantaneous response to a collision with a non-player character (NPC), even when the NPC was inside the field of view. Haptic feedback also enhanced users’ awareness and visual exploration when an NPC approached from the side or back. Furthermore, variation in walking speed was increased by the haptic feedback. These results suggest that the haptic feedback enhances users’ sensitivity to collisions in VR environments.

Pneumatic soft everting robotic structures have the potential to facilitate human transfer tasks due to their ability to grow underneath humans without sliding friction and their utility as a flexible sling when deflated. Tubular structures naturally yield circular cross-sections when inflated, whereas a robotic sling must be both thin enough to grow between a human and their resting surface and wide enough to cradle the human. Recent works have achieved flattened cross-sections by including rigid components into the structure, but this reduces conformability to the human. We present a method of mechanically programming the cross-section of soft everting robotic structures using flexible strips that constrain radial expansion between points along the outer membrane. Our method enables simultaneously wide and thin inflated profiles, and maintains the full multi-axis flexibility of traditional slings when deflated. We develop and validate a model relating geometric design specifications to fabrication parameters, and experimentally characterize their effects on growth rate. Finally, we prototype a soft growing robotic sling system and demonstrate its use for assisting a single caregiver in bed-to-chair patient transfer.

As robots enter everyday spaces like offices, the sounds they create affect how they are perceived. We present “Music Mode,” a novel mapping between a robot’s joint motions and sounds, programmed by artists and engineers to make the robot generate music as it moves. Two experiments were designed to characterize the effect of this musical augmentation on human users. In the first experiment, a robot performed three tasks while playing three different sound mappings. Results showed that participants observing the robot perceived it as more safe, animate, intelligent, anthropomorphic, and likable when playing the Music Mode Orchestra software. To test whether the results of the first experiment were due to the Music Mode algorithm, rather than music alone, we conducted a second experiment. Here, the robot performed the same three tasks, while a participant observed via video, but the Orchestra music was either linked to its movement or random. Participants rated the robots as more intelligent when the music was linked to the movement. Robots using Music Mode logged approximately 200 hours of operation while navigating, wiping tables, and sorting trash, and bystander comments made during this operating time served as an embedded case study. This article has both designerly contributions and engineering contributions. The contributions are as follows: (1) an interdisciplinary choreographic, musical, and coding design process to develop a real-world robot sound feature, (2) a technical implementation for movement-based sound generation, and (3) two experiments and an embedded case study of robots running this feature during daily work activities that resulted in increased likeability and perceived intelligence of the robot.

Skin deformation haptic devices worn on the finger pad provide realistic touch feedback during interactions with virtual objects. Two primary challenges in creating such devices are: first, making a multidegree-of-freedom device (DoF) that is small and lightweight so it does not encumber the wearer and second, providing accurate control of forces displayed to the finger pad. This work presents a 4-DoF finger pad haptic device, called Fourigami, that addresses these challenges. We address the first challenge using origami manufacturing methods and pneumatic actuation to fabricate a 25 g prototype that displays normal, shear, and twist and can be easily worn on the finger pad. We address the second challenge using a low-profile, 6-DoF, force/torque sensor to control forces displayed to the finger. Fourigami has a bandwidth ranging from 2 to 4 Hz depending on direction, and when acting on a human finger, it exerts forces ranging from$\pm$1.0 N in shear, 4.2 N in normal, and$\pm$4.2 N$\cdot$mm of twist. Finally, we demonstrate the device’s efficacy when rendering haptic feedback to a user tracking a sinusoidal trajectory and a trajectory representing interactions with a virtual object.

The navigational abilities of tip-everting soft growing robots, known as vine robots, are compromised when tip-mount devices are added to enable carrying of payloads. We present a new method for securing a vine robot to objects or its environment that exploits the unique eversion-based growth mechanism and flexibility of vine robots, while keeping the tip of the vine robot free of encumbrance. Our implementation is a tip-clutching winch, into which vine robots can insert themselves and anchor to via powerful overlapping belt friction. The device enables passive, high-strength, and reversible fastening, and can easily release the vine robot. This approach enables carrying of loads of at least 28 kg (limited by the tensile strength of the vine robot body material and winch actuator torque capacity), as well as novel material transport and locomotion capabilities.

In this article, we propose a new continuum robotics approach for haptic rendering and comanipulation. This approach is illustrated using a robotic interface with six motorized fixed axes connected by deformable beams, in parallel, to an end effector with 5 degrees of freedom. Apart from the rotation of the motors, this design has no articulation, and the motion of the end effector is achieved by deformation of the beams. The flexible beams are equipped with bending sensors, and the motors have encoders. We use a nonlinear finite element mechanical model of the robot based on a mesh of beam elements that is computed in real time at 20 Hz. The bending sensors are incorporated into the model, which allows us to obtain an accurate estimate of the force exerted by the user on the end effector. The model enables a new methodology for calculating the workspace of the continuum haptic device. The model also is propagated to a higher frequency loop (500 Hz), which performs sensing and control of the robot at high rates, using an admittance-type control to command new positions of the actuators. We show that this control methodology allows haptic rendering of virtual walls that are stiffer than the natural stiffness of the robot. Finally, we demonstrate the use of the device for simple comanipulation tasks.

Soft robot serial chain manipulators with the capability for growth, stiffness control, and discrete joints have the potential to approach the dexterity of traditional robot arms, while improving safety, lowering cost, and providing an increased workspace, with potential application in home environments. This paper presents an approach for design optimization of such robots to reach specified targets while minimizing the number of discrete joints and thus construction and actuation costs. We define a maximum number of allowable joints, as well as hardware constraints imposed by the materials and actuation available for soft growing robots, and we formulate and solve an optimization problem to output a planar robot design, i.e., the total number of potential joints and their locations along the robot body, which reaches all the desired targets, avoids known obstacles, and maximizes the workspace. We demonstrate a process to rapidly construct the resulting soft growing robot design. Finally, we use our algorithm to evaluate the ability of this design to reach new targets and demonstrate the algorithm's utility as a design tool to explore robot capabilities given various constraints and objectives.

To facilitate sensing and physical interaction in remote and/or constrained environments, high-extension, lightweight robot manipulators are easier to transport and reach substantially further than traditional serial chain manipulators. We propose a novel planar 3-degree-of-freedom manipulator that achieves low weight and high extension through the use of a pair of spooling bistable tapes, commonly used in self-retracting tape measures, which are pinched together to form a reconfigurable revolute joint. The pinching action flattens the tapes to produce a localized bending region, resulting in a revolute joint that can change its orientation by cable tension and its location on the tapes though friction-driven movement of the pinching mechanism. We present the design, implementation, kinematic modeling, stiffness behavior of the revolute joint, and quasi-static performance of this manipulator. In particular, we demonstrate the ability of the manipulator to reach specified targets in free space, reach a 2D target with various orientations, and maintain an end-effector angle or stationary bending point while changing the other. The long-term goal of this work is to integrate the manipulator with an aerial robot to enable more capable aerial manipulation.

When humans socially interact with another agent (e.g., human, pet, or robot) through touch, they do so by applying varying amounts of force with different directions, locations, contact areas, and durations. While previous work on touch gesture recognition has focused on the spatio-temporal distribution of normal forces, we hypothesize that the addition of shear forces will permit more reliable classification. We present a soft, flexible skin with an array of tri-axial tactile sensors for the arm of a person or robot. We use it to collect data on 13 touch gesture classes through user studies and train a Convolutional Neural Network (CNN) to learn spatio-temporal features from the recorded data. The network achieved a recognition accuracy of 74% with normal and shear data, compared to 66% using only normal force data. Adding distributed shear data improved classification accuracy for 11 out of 13 touch gesture classes.

Force estimation using neural networks is a promising approach to enable haptic feedback in minimally invasive surgical robots without end-effector force sensors. Various network architectures have been proposed, but none have been tested in real time with surgical-like manipulations. Thus, questions remain about the real-time transparency and stability of force feedback from neural network-based force estimates. We characterize the real-time impedance transparency and stability of force feedback rendered on a da Vinci Research Kit teleoperated surgical robot using neural networks with visiononly, state-only, and state and vision inputs. Networks were trained on an existing dataset of teleoperated manipulations without force feedback. To measure real-time stability and transparency during teleoperation with force feedback to the operator, we modeled a one-degree-of-freedom human and surgeon-side manipulandum that moved the patient-side robot to perform manipulations on silicone artificial tissue over various robot and camera configurations, and tools. We found that the networks using state inputs displayed more transparent impedance than a vision-only network. However, state-based networks displayed large instability when used to provide force feedback during lateral manipulation of the silicone. In contrast, the vision-only network showed consistent stability in all the evaluated directions. We confirmed the performance of the vision-only network for real-time force feedback in a demonstration with a human teleoperator.

While haptics research has traditionally focused on the fingertips and hands, other locations on the body provide large areas of skin that could be utilized to relay large-area haptic sensations. Researchers have thus developed wearable devices that use distributed vibrotactile actuators and distributed pneumatic force displays, but these methods have limitations. In prior work, we presented a novel actuation technique involving stacking pneumatic pouches and evaluated the actuator output. In this work, we developed a wearable haptic device using this actuation technique and evaluated how the actuator output is perceived. We conducted a user study with 20 participants to evaluate users' perception thresholds, ability to localize, and ability to detect differences in contact area and compare their perception using the stacked pneumatic pouch actuation to traditional single-layer pouch actuation. We also used our device with stacked pneumatic actuation in a demonstration of a haptic hug that replicates the dynamics, pressure profile, and mapping to the human back, showcasing how this actuation technique can be used to create novel haptic stimuli.

Relocation of haptic feedback from the fingertips to the wrist has been considered as a way to enable haptic interaction with mixed reality virtual environments while leaving the fingers free for other tasks. We present a pair of wrist-worn tactile haptic devices and a virtual environment to study how various mappings between fingers and tactors affect task performance. The haptic feedback rendered to the wrist reflects the interaction forces occurring between a virtual object and virtual avatars controlled by the index finger and thumb. We performed a user study comparing four different finger-to-tactor haptic feedback mappings and one no-feedback condition as a control. We evaluated users' ability to perform a simple pick-and-place task via the metrics of task completion time, path length of the fingers and virtual cube, and magnitudes of normal and shear forces at the fingertips. We found that multiple mappings were effective, and there was a greater impact when visual cues were limited. We discuss the limitations of our approach and describe next steps toward multi-degree-of-freedom haptic rendering for wrist-worn devices to improve task performance in virtual environments.

Despite non-co-location, haptic stimulation at the wrist can potentially provide feedback regarding interactions at the fingertips without encumbering the user's hand. Here we investigate how two types of skin deformation at the wrist (normal and shear) relate to the perception of the mechanical properties of virtual objects. We hypothesized that a congruent mapping (i.e. when the most relevant interaction forces during a virtual interaction spatially match the haptic feedback at the wrist) would result in better perception than other mappings. We performed an experiment where haptic devices at the wrist rendered either normal or shear feedback during manipulation of virtual objects with varying stiffness, mass, or friction properties. Perception of mechanical properties was more accurate with congruent skin stimulation than noncongruent. In addition, discrimination performance and subjective reports were positively influenced by congruence. This study demonstrates that users can perceive mechanical properties via haptic feedback provided at the wrist with a consistent mapping between haptic feedback and interaction forces at the fingertips, regardless of congruence.

Continuum and soft robots can leverage complex actuator shapes to take onuseful shapes while actuating only a few of their many degrees of freedom. Continuum robotsthat alsogrow increasethe range of potential shapes that can be actuated and enable easier access to constrained environments. Existing models for describing the complex kinematics involved in general actuation of continuum robots rely on simulation or well-behaved stress–strain relationships, but the nonlinear behavior of the thin-walled inflated-beams used in growing robots makes these techniques difficult to apply. Here, we derive kinematic models of single, generally routed tendon paths on a soft pneumatic backbone of inextensible but flexible material from geometric relationships alone. This allows for forward modeling of the resulting shapes with only knowledge of the geometry of the system. We show that this model can accurately predict the shape of the whole robot body and how the model changes with actuation type. We also demonstrate the use of this kinematic model for inverse design, where actuator designs are found based on desired final robot shapes. We deploy these designed actuators on soft pneumatic growing robots to show the benefits of simultaneous growth and shape change.

Knowledge of interaction forces during teleoperated robot-assisted surgery could be used to enable force feedback to users and evaluate tissue handling skill. However, direct force sensing at the end-effector is challenging because it requires biocompatible, sterilizable, and cost-effective sensors. Vision-based neural networks are a promising approach for providing useful force estimates, though questions remain about generalization to new scenarios and real-time inference. We present a force estimation neural network that uses RGB images and robot state as inputs. Using a self-collected dataset, we compared the network to variants that included only a single input type, and evaluated how they generalized to new viewpoints, workspace positions, materials, and tools. We found that the vision-only network was sensitive to shifts in viewpoints, while networks with state inputs were sensitive to vertical shifts in workspace. The network with both state and vision inputs had the highest accuracy for an unseen tool, while the state-only network was most accurate for an unseen material. Through feature removal studies, we found that using only force features produced better accuracy than using only kinematic features as input. The network with both state and vision inputs outperformed a physics-based model in accuracy for seen material. It showed comparable accuracy but faster computation times than a recurrent neural network, making it better suited for real-time applications.

Pneumatic wearable haptic devices can provide distributed pressure feedback to human operators during robot teleoperation and in virtual and augmented reality. However, these devices have an inherent trade-off between the spatial coverage of their pressure output and their resolution and dynamic response. To achieve specified spatial resolution and dynamic response, we propose a macro-mini actuation approach that stacks a number of smaller inflatable pouches atop a larger inflatable pouch. We develop models for the static and dynamic responses of single and stacked pouches and compare these with experimental results, providing guidelines for the design of wearable stacked pneumatic displays. Finally, we demonstrate this pneumatic macro-mini approach by replicating the time series pressure profiles of data collected from a huggable robot embedded with distributed force sensors.

Simulating soft robots in cluttered environments remains an open problem due to the challenge of capturing complex dynamics and interactions with the environment. Furthermore, fast simulation is desired for quickly exploring robot behaviors in the context of motion planning. In this paper, we examine a particular class of inflated-beam soft growing robots called "vine robots," and present a dynamics simulator that captures general behaviors, handles robot-object interactions, and runs faster than real time. The simulator framework uses a simplified multi-link, rigid-body model with contact constraints. To bridge the sim-to-real gap, we develop methods for fitting model parameters based on video data of a robot in motion and in contact with an environment. We provide examples of simulations, including several with fit parameters, to show the qualitative and quantitative agreement between simulated and real behaviors. Our work demonstrates the capabilities of this high-speed dynamics simulator and its potential for use in the control of soft robots.

Inflated continuum robots are promising for a variety of navigation tasks, but controlling their motion with a small number of actuators is challenging. These inflated beam robots tend to buckle under compressive loads, producing extremely tight local curvature at difficult-to-control buckle point locations. In this paper, we present an inflated beam robot that uses distributed stiffness changing sections enabled by positive pressure layer jamming to control or prevent buckling. Passive valves are actuated by an electromagnet carried by an electromechanical device that travels inside the main inflated beam robot body. The valves themselves require no external connections or wiring, allowing the distributed stiffness control to be scaled to long beam lengths. Multiple layer jamming elements are stiffened simultaneously to achieve global stiffening, allowing the robot to support greater cantilevered loads and longer unsupported lengths. Local stiffening, achieved by leaving certain layer jamming elements unstiffened, allows the robot to produce "virtual joints" that dynamically change the robot kinematics. Implementing these stiffening strategies is compatible with growth through tip eversion and tendonsteering, and enables a number of new capabilities for inflated beam robots and tip-everting robots.

Rich haptic sensory feedback in response to user interactions is desirable for an effective, immersive virtual reality or teleoperation system. However, this feedback depends on material properties and user interactions in a complex, non-linear manner. Therefore, it is challenging to model the mapping from material and user interactions to haptic feedback in a way that generalizes over many variations of the user's input. Current methodologies are typically conditioned on user interactions, but require a separate model for each material. In this paper, we present a learned action-conditional model that uses data from a vision-based tactile sensor (GelSight) and user's action as input. This model predicts an induced acceleration that could be used to provide haptic vibration feedback to a user. We trained our proposed model on a publicly available dataset (Penn Haptic Texture Toolkit) that we augmented with GelSight measurements of the different materials. We show that a unified model over all materials outperforms previous methods and generalizes to new actions and new instances of the material categories in the dataset.

Soft growing robots are proposed for use in applications such as complex manipulation tasks or navigation in disaster scenarios. Safe interaction and ease of production promote the usage of this technology, but soft robots can be challenging to teleoperate due to their unique degrees of freedom. In this paper, we propose a human-centered interface that allows users to teleoperate a soft growing robot for manipulation tasks using arm movements. A study was conducted to assess the intuitiveness of the interface and the performance of our soft robot, involving a pick-and-place manipulation task. The results show that users were able to complete the task 97% of the time and achieve placement errors below 2 cm on average. These results demonstrate that our body-movement-based interface is an effective method for control of a soft growing robot manipulator.

Pneumatically operated soft growing robots that extend via tip eversion are well-suited for navigation in confined spaces. Adding the ability to interact with the environment using sensors and tools attached to the robot tip would greatly enhance the usefulness of these robots for exploration in the field. However, because the material at the tip of the robot body continually changes as the robot grows and retracts, it is challenging to keep sensors and tools attached to the robot tip during actuation and environment interaction. In this paper, we analyze previous designs for mounting to the tip of soft growing robots, and we present a novel device that successfully remains attached to the robot tip while providing a mounting point for sensors and tools. Our tip mount incorporates and builds on our previous work on a device to retract the robot without undesired buckling of its body. Using our tip mount, we demonstrate two new soft growing robot capabilities: (1) pulling on the environment while retracting, and (2) retrieving and delivering objects. Finally, we discuss the limitations of our design and opportunities for improvement in future soft growing robot tip mounts.

In this article, we present a planning framework that uses a combination of implicit (robot motion) and explicit (visual/audio/haptic feedback) communication during mobile robot navigation. First, we developed a model that approximates both continuous movements and discrete behavior modes in human navigation, considering the effects of implicit and explicit communication on human decision-making. The model approximates the human as an optimal agent, with a reward function obtained through inverse reinforcement learning. Second, a planner uses this model to generate communicative actions that maximize the robot's transparency and efficiency. We implemented the planner on a mobile robot, using a wearable haptic device for explicit communication. In a user study of an indoor human-robot pair orthogonal crossing situation, the robot is able to actively communicate its intent to users in order to avoid collisions and facilitate efficient trajectories. Results show that the planner generated plans that are easier to understand, reduce users` effort, and increase users' trust of the robot, compared to simply performing collision avoidance. The key contribution of this article is the integration and analysis of explicit communication (together with implicit communication) for social navigation.

Haptic shape displays provide compelling touch interactions by allowing users to freely explore a rendered surface. However, these displays are currently limited to 2.5-D surfaces due to the space requirements of their actuation. Building on previous work in haptic jamming, we developed a novel, soft 3-D shape display. A fully 3-D display that a user can grasp and hold allows for improved interactions for applications such as medical palpation training and virtual reality experiences. The shape display is implemented as an inflatable silicone membrane with embedded particle jamming cells that change stiffness and soft pneumatic actuators that control the distance between points on the surface. The device was modeled as a mass-spring system, and this model is used to develop a control sequence for a device to match a target shape. Due to constraints in actuation imposed by the 3-D geometry of the device, we developed an automatic design algorithm for the display, so that a display can be custom-designed to reach a set of target shapes using a relatively small number of actuators.

Soft robotics and pneumatic actuation present opportunities for lightweight wearable haptic devices that provide distributed touch feedback to the skin. Ideally, such devices would be easily donned and doffed, since permanent coverage of a large area of the skin is undesirable. Here we present the design and evaluation of a concept device called HapWRAP: a growing haptic device constructed from flexible low density polyethylene. Controlled air flow through tubes and pouches allows HapWRAP to grow out of a compact housing unit and provide a combination of directional and force feedback to a user. When activated, HapWRAP grows up and around the forearm; its loops form a temporary sleeve. After growth, pneumatic actuators inflate and deflate to stimulate mechanoreceptors in the skin at distinguishable locations. This paper describes the design and manufacturing of HapWRAP, reports its performance metrics, and tests its suitability as a haptic feedback device. Participants were able to interpret force and direction cues from HapWRAP with 92.5% accuracy. These findings suggest that HapWRAP can be successfully used for applications where both force and direction cues are necessary.

Fully autonomous navigation in populated environments is still a challenging problem for mobile robots. This paper explores the idea of using active human-robot communication to facilitate navigation tasks. We propose to convey a robot's intent to human users via a wearable haptic interface. The interface can display distinct haptic cues by modulating vibration amplitudes and patterns. We applied the concept to a single human/single robot orthogonal encounter scenario, where one of the two parties has to yield the right of way to avoid collision. Under certain conditions, the robot's intent (to yield to the human or not) is revealed to the human via the haptic interface prior to the interaction. We conducted an experiment with 10 users, in which the robot was teleoperated as a substitute for autonomy. Results show that, when given priority, users become more risk-accepting and use different strategies to navigate the collision scenario than when the robot takes priority or there is no haptic communication channel. In addition, we propose a social-force based model to predict human movement during navigation. The effect of communication can be explained as a shift in the user's safety buffer and expectation of the robot's future velocity.

Hands-on training is an indispensable part of surgical practice. As the tools used in the operating room become more intricate, the demand for efficient training methods increases. This work proposes a robotic assistance-as-needed method for training with surgical teleoperated robots. The method adapts the intensity of the assistance according to the trainee's current and past performance while gradually increasing the level of control of the trainee as the training progresses. The work includes an experiment comprising 160 acquisition sessions from 16 novice subjects performing a bimanual teleoperated exercise with a da Vinci Research Kit surgical console. Results capture the subtleties in the task's learning curve with and without robotic assistance and hint at the potential of robotic assistance for complex visuomotor training. Although robotic assistance for motor learning has received mixed results that range from beneficial to detrimental effects, this study shows such assistance may increase the rate of learning of certain skills in complex motor tasks.

For many types of robots, avoiding obstacles is necessary to prevent damage to the robot and environment. As a result, obstacle avoidance has historically been an important problem in robot path planning and control. Soft robots represent a paradigm shift with respect to obstacle avoidance because their low mass and compliant bodies can make collisions with obstacles inherently safe. Here we consider the benefits of intentional obstacle collisions for soft robot navigation. We develop and experimentally verify a model of robot-obstacle interaction for a tip-extending soft robot. Building on the obstacle interaction model, we develop an algorithm to determine the path of a growing robot that takes into account obstacle collisions. We find that obstacle collisions can be beneficial for open-loop navigation of growing robots because the obstacles passively steer the robot, both reducing the uncertainty of the location of the robot and directing the robot to targets that do not lie on a straight path from the starting point. Our work shows that for a robot with predictable and safe interactions with obstacles, target locations in a cluttered, mapped environment can be reached reliably by simply setting the initial trajectory. This has implications for the control and design of robots with minimal active steering.

Teleoperation systems could benefit from force sensory substitution when kinesthetic force feedback systems are too bulky or expensive, and when they cause instability by magnifying force feedback. We aim to magnify force feedback using sensory substitution via force-controlled tactile skin deformation, using a device with the ability to provide tangential and normal force directly to the fingerpads. The sensory substitution device is able to provide skin deformation force feedback over ten times the maximum stable kinesthetic force feedback on a da Vinci Research Kit teleoperation system. We evaluated the effect of this force magnification in two experimental tasks where the goal was to minimize interaction force with the environment. In a peg transfer task, magnified force feedback using sensory substitution improved participants' performance for force magnifications up to ten times, but decreased performance for higher force magnifications. In a tube connection task, sensory substitution that doubled the force feedback maximized performance; there was no improvement at the larger magnifications. These experiments demonstrate that magnified force feedback using sensory substitution via force-controlled skin deformation feedback can decrease applied forces similarly to magnified kinesthetic force feedback during teleoperation.

As the field of haptics in virtual reality expands, wearable devices are being explored as alternatives to traditional kinesthetic force feedback devices, which are often limited in workspace. Skin deformation feedback offers a user-grounded feedback modality that mimics cutaneous interactions with the real world but can suffer from force-output saturation due to the actuation constraints required to achieve a small form factor. Saturation of haptic devices limits the mechanical properties and interactions that can be rendered in a virtual environment, specifically the weight that can be rendered when a user manipulates a virtual object. We use scaled inertial forces to alter virtual weight perception during a dynamic grasp-lift-and-place task in a virtual environment with haptic feedback via two wearable skin deformation feedback devices. A study was conducted, beginning with an open response exercise to assess how participants interpreted scaled inertial forces when interacting with virtual blocks. Participants then performed a series of trials to measure the Point of Subjective Equality of virtual weight with scaled inertial forces, using a reference of 200 g under the normal (no scaling) condition. PSEs for inertial scaling factors of 2 and 3 were 171 g and 151 g, respectively. These results demonstrate the effectiveness of a unique haptic rendering algorithm that can convey larger weights without saturating the force output of the haptic device.

We present a pneumatic actuator capable of changing length by 1000%, applying both pushing and pulling forces, and independently modulating its length and stiffness. These characteristics are enabled by individually addressable internal and external chambers that work antagonistically against one another. The high deformation with low hysteresis is achieved by wrinkling of thin materials that are assumed to be inextensible but flexible, as opposed to stretchable. A model for the actuator is presented and validated with experimental results, showing capabilities of high strain, pushing and pulling, and independent control of length and stiffness. These characteristics are motivated by the application of a compliant truss robot. Accordingly, we show a simple grounded tetrahedron with three actuator elements and three static elements. We demonstrate motion of the tetrahedron apex against external loads and the ability of the structure to vary its stiffness. The actuator offers a unique set of characteristics that could increase the capabilities of soft robotic devices.

Adapting robot actions to human motions can make human-robot interactions (HRI) more effective. Here, we aim to optimize guidance from haptic devices based on a user's response to produce better task performance. We used Gaussian processes to model the motions a human user made in response to applied torques from an ungrounded control moment gyroscope haptic device. We then used Gaussian process dynamic programming to generate optimized haptic cues to guide the user to rotate the device toward 3D targets. We compared the performance of naive and optimized policies in simulations and with a human user, and found that dynamic programming can significantly improve haptic guidance in cases where human responses are highly variable or inconsistent with the cued haptic direction.

In this article, we present a bi-directional communication scheme that facilitates interaction between a person and a mobile robot that follows the person. A person-following robot can assist people in many applications including load carrying, elder care, and emotional support. However, commercially available personal robot systems usually have limited sensing and actuation capabilities. They are not expected to function perfectly in complex environments, and human intervention is required when the robot fails. We propose to use a holdable mechatronic device to reduce the user’s effort in communication and enable natural interaction during the intervention. Our design of the holdable device consists of two parts: a haptic interface that displays touch cues to convey the robot’s failure status via asymmetric vibrations, and a command interface for teleoperating the robot follower with hand gestures. We experimentally evaluated the device and the communication strategy in two sets of user studies with a controlled environment and a physical robot follower. Results show that with the proposed method, users are able to perform their tasks better, respond to robot failure events faster, and adjust walking speed according to the robot’s limitations. We also demonstrate that users can successfully teleoperate the robot to avoid obstacles when navigating in challenging environments.

editorial Open AccessHaptic Dimensions of Human-Robot Interaction Share on Author: Allison M. Okamura Department of Mechanical Engineering, Stanford University, Stanford, CA, USA Department of Mechanical Engineering, Stanford University, Stanford, CA, USAView Profile Authors Info & Affiliations ACM Transactions on Human-Robot InteractionVolume 7Issue 1May 2018 Article No.: 6pp 1–3https://doi.org/10.1145/3209768Published:16 May 2018 2citation646DownloadsMetricsTotal Citations2Total Downloads646Last 12 Months155Last 6 weeks14 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Haptic force feedback systems are unique in their ability to dynamically render physical representations. Although haptic devices have shown promise for supporting learning, prior work mainly describes results of haptic-supported learning without identifying underlying learning mechanisms. To this end, we designed a haptic-supported learning environment and analyzed four students who used it to make connections between two different mathematical representations of sine and cosine: the unit circle, and their graph on the Cartesian plane. We highlight moments where students made connections between the representations, and identify how the haptic feedback supported these moments of insight. We use this evidence in support of a proposed theoretical and design framework for educational haptics. This framework captures four types of haptic representations, and focuses on one -- the haptic bridge -- that effectively scaffolds sense-making with multiple representations.

WAVES, a Wearable Asymmetric Vibration Excitation System, is a novel wearable haptic device for presenting three dimensions of translation and rotation guidance cues. In contrast to traditional vibration feedback, which usually requires that users learn to interpret a binary cue, asymmetric vibrations have been shown to induce a pulling sensation in a desired direction. When attached to the fingers, a single voicecoil actuator presents a translation guidance cue and a pair of voicecoil actuators presents a rotation guidance cue. The directionality of mechanoreceptors in the skin led to our choice of the location and orientation of the actuators in order to elicit very strong sensations in certain directions. For example, users distinguished a "left" cue versus a "right" cue 94.5% of the time. When presented with one of six possible direction cues, users on average correctly identified the direction of translation cues 86.1% of the time and rotation cues 69.0% of the time.

One of the main barriers to immersivity during object manipulation in virtual reality is the lack of realistic haptic feedback. Our goal is to convey compelling interactions with virtual objects, such as grasping, squeezing, pressing, lifting, and stroking, without requiring a bulky, world-grounded kinesthetic feedback device (traditional haptics) or the use of predetermined passive objects (haptic retargeting). To achieve this, we use a pair of finger-mounted haptic feedback devices that deform the skin on the fingertips to convey cutaneous force information from object manipulation. We show that users can perceive differences in virtual object weight and that they apply increasing grasp forces when lifting virtual objects as rendered mass is increased. Moreover, we show how naive users perceive changes of a virtual object's physical properties when we use skin deformation to render objects with varying mass, friction, and stiffness. These studies demonstrate that fingertip skin deformation devices can provide a compelling haptic experience appropriate for virtual reality scenarios involving object manipulation.

We describe a new series pneumatic artificial muscle (sPAM) and its application as an actuator for a soft continuum robot. The robot consists of three sPAMs arranged radially round a tubular pneumatic backbone. Analogous to tendons, the sPAMs exert a tension force on the robot's pneumatic backbone, causing bending that is approximately constant curvature. Unlike a traditional tendon driven continuum robot, the robot is entirely soft and contains no hard components, making it safer for human interaction. Models of both the sPAM and soft continuum robot kinematics are presented and experimentally verified. We found a mean position accuracy of 5.5 cm for predicting the end-effector position of a 42 cm long robot with the kinematic model. Finally, closed-loop control is demonstrated using an eye-in-hand visual servo control law which provides a simple interface for operation by a human. The soft continuum robot with closed-loop control was found to have a step-response rise time and settling time of less than two seconds.

Wearable haptic feedback devices for virtual reality, human-robot interaction, and motion guidance require lightweight actuators that display clearly discernible cues to the user. These goals motivate the design of WRAP, a wearable, pneumatically actuated haptic feedback device. WRAP displays a variety of tactile sensations to the user by inflating a thermoplastic pneumatic actuator in direct contact with the skin. This paper describes the design and construction of WRAP, shows its effectiveness in indicating direction cues to users, and demonstrates two other appUcations for WRAP in image-guided medical interventions and human-computer interaction. Users were able to identify translation and rotation cues from WRAP with 99.4% accuracy. Our results suggest that WRAP is suitable for a variety of wearable and portable applications in which direction cues are beneficial.

Stroke is the leading cause of adult disability. Many robots have been developed to administer movement therapies or provide physical assistance to stroke survivors suffering from movement deficits. One effective approach has been to support the weight of the arm, offloading shoulder abductor muscles that have become coupled to elbow muscles. However, patients have limited access to such robots due to the robots' complexity, cost, and bulk. To counter this problem, we developed a lightweight (350 g), inexpensive external actuator, which we call an exomuscle. We constructed a prototype exomuscle by reinforcing a plastic bladder with a fabric bag that is sewn to supporting straps. The bladder can then be inflated with pressurized air to provide expansive forces between the user's torso and arm, supporting shoulder abduction. A seam acting as a hinge joint connects the exomuscle to the torso. We demonstrate that our exomuscle reduces muscular effort by 74% in isometric tasks and 72% in dynamic reaching tasks while minimally affecting the range of motion of the shoulder and elbow (average 4% reduction) on three users ranging from 165 to 188 cm tall. Future studies will evaluate the exomuscle with users who have post-stroke motor impairments.

Many robotic systems achieve position sensing through the use of optical encoders that specify the position of a joint to a certain resolution. Encoders effectively quantize joint space coordinates and introduce position measurement error in the process. This error propagates to operational space coordinates, limiting end-effector position and orientation resolution, and also to joint and operational space coordinate derivatives, manifesting as noise that can vitiate the signal. In this paper, we characterize encoder error in a robotic system. Given encoder specifications, robot kinematics, and discrete transfer functions mapping coordinates to their derivatives, we describe the propagation of quantization error on joint space coordinates to operational space coordinates, joint space coordinate derivatives, and operational space coordinate derivatives. We establish two results for quantization error. The first is a general result useful for establishing worst-case bounds. The second models each quantization as independent additive pseudo quantization noise (PQN) for which stochastic metrics on the error are determined. Experimental data gathered from a Phantom Premium robot/haptic device supports the analytical results.

Surgeries involving interaction with soft tissue like the brain need to minimize shear and normal forces that can cause tissue damage or hemorrhage. Other surgeries require the ability to follow a complex, curved path, such as through an intestine or to a kidney stone. This paper presents a soft catheter that has the potential to aid in these challenging cases. The soft catheter is capable of apical extension in which the tip extends while the rest of the catheter remains stationary. This limits shear forces with the environment, easing movement of a body's tip through a constrained space. The soft catheter is pre-formed to patient-specific trajectories, meaning that normal forces against tissue would only arise due to errors between the actual and desired paths; we show decrease in normal force applied to the environment on the order of 100 compared to a standard catheter in a 30 degree bend. Setting the internal pressure allows for control of catheter stiffness, with a 500 times difference over the range of tested pressures. Manual operation to reach a surgical site requires only holding the correct orientation at the entry point into the body and setting the internal pressure of the catheter. This soft catheter could offer two benefits: the ability to apply low tissue interaction forces and reach challenging locations within the body.

Haptic Jamming, the approach of simultaneously controlling mechanical properties and surface deformation of a tactile display via particle jamming and pneumatics, shows promise as a tangible, shape-changing human-computer interface. Previous research introduced device design and described the force-displacement interactions for individual jamming cells. The work in this article analyzes the shape output capabilities of a multi-cell array. A spring-mass deformable body simulation combines models of the three actuation inputs of a Haptic Jamming surface: node pinning, chamber pressurization, and cell jamming. Surface measurements of a 12-cell prototype from a depth camera fit the mass and stiffness parameters to the device during pressurization tests and validate the accuracy of the model for various actuation sequences. The simulator is used to develop an algorithm that generates a sequence of actuation inputs for a Haptic Jamming array of any size in order to match a desired surface output shape. Data extracted from topographical maps and three-dimensional solid object models are used to evaluate the shape-matching algorithm and assess the utility of increasing array size and resolution. Results show that a discrete Laplace operator applied to the input is a suitable predictor of the correlation coefficient between the desired shape and the device output.

We present the inverse pneumatic artificial muscle (IPAM), a new soft actuator that is powered by pneumatics in a manner inverse to traditional pneumatic muscles: low pressure, rather than high, contracts the muscle. The IPAM improves on the 50-year-old standard in soft pneumatic actuators, the McKibben muscle, but retains many of the advantages that have drawn roboticists to this artificial muscle over the years. The McKibben muscle produces up to 40% strain, and has nonlinear control with friction and hysteresis, whereas the IPAM attains strains of over 300% and has a nearly linear mapping between input pressure and force/length output and no sliding friction. Crucially, the IPAM retains the soft structure, low weight, compliance, and robustness that the McKibben muscle boasts. We present a simple model to describe the behavior of the muscle, as well as force, displacement, pressure, and speed tests validating the model and characterizing the IPAM's performance. Further, we present two practical implementations using the IPAM: an active brace and a robotic finger.

Increasing the level of autonomy in robot-assisted surgery has the potential to improve the safety, speed, and applicability of robot-assisted surgical systems. To facilitate the development and incorporation of robot autonomy in clinical settings, human-robot collaboration models have been suggested in which human and robotic agents work together to accomplish a task. In this work, we measure performance of several human-robot collaboration models in two experiments based on the task of segmenting a stiff inclusion in soft tissue, which simulates a tumor. In the inclusion segmentation experiment, twelve participants explored an artificial tissue and identified the inclusion boundary under the collaboration models of (1) teleoperation, (2) supervised control, (3) traded control, and (4) full autonomy. In the boundary identification experiment, we isolate the performance of human and robotic agents in the boundary identification sub-task; participants and a robotic agent independently identified the boundary of four virtually palpated tissues. Results from the inclusion segmentation experiment indicate that human agents complete the task faster; teleoperation had the fastest task times. Results of both experiments indicate that the robotic agent identifies boundaries with higher sensitivity and less variance than human agents. This indicates that task accuracy increases when a robotic agent segments the boundary, while including a human agent can decrease the overall task time.

A Haptic Jamming tactile display, consisting of an array of thin particle jamming cells, can change its shape and mechanical properties simultaneously through a combination of vacuuming individual cells, pressurizing the air chamber beneath the surface, and pinning the nodes between the cells at various heights. In previous particle jamming devices for haptics and soft robotics, shape has typically been commanded open-loop or manipulated directly by a human user. A new algorithm was designed for the three types of actuation inputs for a Haptic Jamming surface, using the depth map provided by an RGB-D sensor as shape feedback for closed-loop control to match a desired surface input. To test the closed-loop control accuracy of the system, a mass-spring model of the Haptic Jamming device generated three unique surface shapes as desired inputs into the controller with a mean height range of 25.1 mm. The average correlation coefficient between the desired input shape and the experimental output generated by the controller on the actual device across four trials for each shape was 0.88, with an average height error of 2.7 mm. When the desired input surface is generated from a 3D model of an object that a Haptic Jamming surface cannot necessarily re-create, the difference between the input and the output increases substantially. However, simulation of a larger array suggests that a Haptic Jamming surface can provide a compelling match for these more complicated shapes.

Virtual and augmented reality systems that immerse users in a 3D environment could benefit from haptic (force and/or tactile) feedback to increase realism and task performance. A canonical task that cannot be achieved compellingly without haptic feedback is the grasp and lift of an object with mass. Traditional kinesthetic (force) feedback devices provide realistic physical interactions with virtual objects, but typically require large actuators, making them bulky and encumbering. In contrast, tactile feedback devices can use smaller actuators and may enable freer movement of a user in a virtual world. One form of tactile feedback, skin deformation feedback, has been previously shown to provide effective haptic feedback of force, stiffness, and friction. Here we compare human perception of mass via kinesthetic feedback and skin deformation feedback in a virtual environment. Participants grasp and lift two virtual blocks and attempt to equalize their masses by the method of adjustments. From the accuracy of this equalization, we determined that the Weber Fraction (the just noticeable difference in proportion to the original stimulus value) of virtual mass during a grasp and lift task was 0.11 for kinesthetic force feedback and 0.35 for skin deformation feedback. In addition, participants exhibited differences in exploratory procedures between the two types of feedback.

In robot-assisted surgery, exploration and manipulation tasks can be achieved through collaboration among robotic and human agents. Collaboration models can potentially include multiple agents working towards a shared objective-a scenario referred to as multilateral manipulation. We present a flexible software framework to expedite development of various multilateral manipulation strategies. We demonstrate the effectiveness of an implementation of the framework in a palpation task. Five different collaboration models were tested in which the goal of the multilateral manipulation system is to segment a stiff inclusion from its surrounding soft tissue: three of these collaboration models used machine learning methods for segmentation, and two required human operator segmentation. The collaboration models tested were: 1) fully autonomous exploration of the tissue; 2) shared control between a human and robotic agent; 3) supervised control where the operator dictates commands to the robot; 4) traded control between the two agents; and 5) bilateral teleoperation. Results indicate tradeoffs in sensitivity, maximum force applied, safety implications, and duration of experiment among the five models.

Concentric tube surgical robots are minimally invasive devices with the advantages of snake-like reconfigurability, long and thin form factor, and placement of actuation outside the patient's body. These robots can also be designed and manufactured to acquire targets in specific patients for treating specific diseases in a manner that minimizes invasiveness. We propose that concentric tube robots can be manufactured using 3-D printing technology on a patient- and procedure-specific basis. In this paper, we define the design requirements and manufacturing constraints for 3-D printed concentric tube robots and experimentally demonstrate the capabilities of these robots. While numerous 3-D printing technologies and materials can be used to create such robots, one successful example uses selective laser sintering to make an outer tube with a polyether block amide and uses stereolithography to make an inner tube with a polypropylene-like material. This enables a tube pair with precurvatures of 0.0775 and 0.0455 mm-1, which can withstand strains of 20% and 5.5% for the outer and inner tubes, respectively.

During robot-assisted minimally invasive surgery, surgeons perform challenging dexterous tasks, including the manipulation of soft tissue and suture tying. In the absence of environment force sensing of tool-tissue interaction forces to provide force feedback, surgeons must rely on visual feedback to modulate the grip force they apply on the environment. Clinical systems, like the da Vinci Surgical System (Intuitive Surgical, Inc.), use physical springs to provide closing resistance on the gripper degree-of-freedom (DOF) of the master manipulator. This feedback provides increasing force resistance as the gripper is closed. To determine the effect of master manipulator gripper stiffness on performance in a teleoperated manipulation task, we designed a new and open source gripper, the OmniGrip. The OmniGrip attaches to a SensAble Phantom Omni (now available as Geomagic Touch), replacing the stySensAble Phantom Omnilus end effector, and providing the ability for user programmable force characteristics. We conducted a study in which participants used an OmniGrip to teleoperate a Raven II surgical robRaven II surgical robotic systemotic system in a pick-and-place task. Increasing the stiffness of the OmniGrip resulted in reduced interaction forces at the slave-side environment. Additionally, these interaction forces were significantly lower when the OmniGrip as compared to when using the Phantom Omni stylus.

During multi-lateral collaborative teleoperation, where multiple human or autonomous agents share control of a teleoperation system, it is important to be able to convey individual user intent. One option for conveying the actions and intent of users or autonomous agents is to provide force guidance from one user to another. Under this paradigm, forces would be transmitted from one user to another in order to guide motions and actions. However, the use of force guidance to convey intent can mask environmental force feedback. In this paper we explore the possibility of using tactile feedback, in particular skin deformation feedback, skin deformation feedback to convey collaborative intent while preserving environmental force perception. An experiment was performed to test the ability of participants to use force guidance and skin deformation guidance to follow a path while interacting with a virtual environment. In addition, we tested the ability of participants to discriminate virtual environment stiffness when receiving either force guidance or skin deformation guidance. We found that skin deformation guidance resulted in a reduction of path-following accuracy, but increased the ability to discriminate environment stiffness when compared with force feedback guidance.

Robotic needle steering systems for minimally invasive medical procedures require complementary medical imaging systems to track the needles in real time. Ultrasound is a promising imaging modality because it offers relatively low-cost, real-time imaging of the needle. Previous methods applied vibration to the base of the needle using a voice coil actuator, in order to make the needle visible in power Doppler ultrasound. We propose a new method for needle tip vibration, using electromagnetic actuation of small permanent magnets placed inside the needle to improve needle tip visibility in power Doppler imaging. Robotic needle insertion experiments using artificial tissue and ex vivo porcine liver showed that the electromagnetic tip vibration method can generate a stronger Doppler response compared to the previous base vibration method, resulting in better imaging at greater needle depth in tissue. It also eliminates previous issues with vibration damping along the shaft of the needle.

In a teleoperated system, misalignment between the master and slave manipulators can result from clutching, errors in the kinematic model, and/or sensor errors. This study examines the effects of type and magnitude of misalignment on the performance of the teleoperator. We first characterized the magnitude and direction of orientation misalignment created when clutching and unclutching during use of two surgical robots: the Raven II and the da Vinci Research Kit. We then purposely generated typical misalignments in order to measure the impact of such misalignment on user performance of a peg transfer task with the Raven II. Users were able to compensate for misalignment angles up to approximately 20 degrees in both tool orientation and camera viewpoint misalignment. These results can be used to guide the design and control of teleoperated systems for a variety of applications.

During robotic teleoperation, the dynamics of the master manipulators and the control of remote-side instruments impose challenges on the motor system of the human operator, and may impact performance and learning. In teleoperated robot-assisted minimally invasive surgery, there is a clear correlation between patient outcomes and the surgeon's case experience. However, the effect of the teleoperator on human motor skills and the relationship between these motor skills and patient outcomes are unknown. We used the da Vinci Research Kit, a custom research version of the da Vinci Surgical System, to compare teleoperated and open needle-driving movements of experienced da Vinci surgeons and novices. The experimental protocol consisted of structured but unconstrained needle driving trials repeated 80 times to allow for computational modeling of movement coordination and learning. Kinematic analysis showed that teleoperation increases trial time but reduces path length, that the trial times and path lengths of experienced surgeons are smaller than those of novices. In addition, there are significant differences in learning between experienced surgeons and novice users. Modeling of the movements and learning processes of experienced and novice surgeons may be used in the design of novel controllers that will expand robotic surgery capabilities and improve robot-assisted surgical skill acquisition.

When a person interacts with an environment using a tool, he/she receives tactile information in the form of fingerpad skin deformation. Different interaction forces and torques on the tool cause different skin deformation patterns on the fingerpads. We designed a 6-degree-of-freedom tactile device that creates similar skin deformation patterns on the fingerpads. The device communicates force and torque information by translating and rotating skin deformation tactors relative to the fingerpads. An experiment was conducted to determine participants' ability to use skin deformation tactile cues to perform a peg-in-hole insertion task. Results show that participants can use the tactile cues to reduce interaction force and torque, and they use the tactile force cues to reduce interaction force more than they use the tactile torque cues to reduce interaction torque. Rendering force and torque cues simultaneously causes device saturation and degrades user performance. These results suggest that additional training may help participants use the skin deformation torque cues, and motivate a tactile device design that decouples force and torque skin deformation rendering to minimize device saturation. Fingerpad skin deformation is a promising form of tactile feedback to convey force and torque information in teleoperation systems such as robot-assisted surgery, where force feedback may be undesirable due to stability and safety concerns.

Robotic systems, such as prosthetics and exoskeletons, offer people suffering from motor impairments a chance to regain lost physical functionality. However, the neural control that individuals are able to exert over these robots is currently limited. This is due to both lack of control authority in many degrees of freedom and insufficient sensory feedback through the human-robot interface. We propose that haptic feedback is paramount for accurate and efficient control of robots via brain-computer interfaces (BCIs). Skin stretch at the fingertip is a novel form of haptic feedback for improving BCI-based robot control. In this paper, we describe the design of a BCI-driven skin-stretch device, assess several control paradigms for this device, and evaluate its effectiveness in a small user study. We show that BCI-based movement-intent classification improved in the presence of skin-stretch feedback for 3 of 4 healthy individuals controlling a computer cursor via an inexpensive, commercial electroencephalography-based (EEG) BCI.

In robot-assisted surgery, manipulation tasks can be achieved through collaboration among robotic and human agents. Collaboration models can potentially include multiple agents working towards a shared objective - a scenario referred to as multilateral manipulation. In this work, we examine multilateral manipulation in the task of debridement: removing dead tissue or damaged tissue fragments to allow remaining healthy tissue to heal. We extended our previously developed multilateral manipulation software framework to the task of debridement and implemented four different collaboration models: (1) fully autonomous debridement, (2) shared control between a human and robotic agent, (3) supervised control where the operator identifies foreign bodies to be excised, and (4) teleoperation. We demonstrate these collaboration models with the RAVEN-II robot, an open-architecture surgical robot with two cable-driven 7-DOF arms. Each collaboration model included methods to remove foreign bodies from the field of view of a stereoscopic camera setup. Results indicate tradeoffs in experiment duration, hardware requirements, and safety implications among the four collaboration models.

Automation of repetitive tasks can improve laparoscopic surgical procedures by unloading surgeons and reducing duration, trauma, and expense. However, surgical procedures involve delicate manipulation of deformable tissues in a very dynamic environment, suggesting that automated execution of surgical tasks should be carried out under the supervision of the surgeon. We propose a teleoperated architecture that allows a surgeon to employ and supervise agents that can autonomously perform or assist with surgical tasks. The architecture is independent of the automation method. It includes a dominance factor that allows the surgeon to take control over the slave robot at any time, and an aggressiveness factor that sets the performance pace of the autonomous agent. We tested the architecture during execution of a multilateral tension-and-cut task, where a human operator and an autonomous agent are responsible for tensioning or cutting of a tissue. The architecture allowed for supervised and paced automation of the task. We found that collaboration of the human operator and autonomous agent can lead to shorter completion time compared to performance of only a human.

When we use a tool to explore or manipulate an object, friction between the surface of the tool and the fingerpads generates skin stretch cues that are related to the interaction forces between the tool and the object. In this study, we emulate these naturally occurring skin stretch cues in order to convey force direction and magnitude information to users during teleoperation. We hypothesize that skin stretch feedback is a useful substitute for kinesthetic force feedback in force-sensitive teleoperated tasks. In this study, ten participants performed teleoperated palpation to determine the orientation of a stiff region in a surrounding artificial tissue using five feedback conditions: skin stretch, force, reduced gain force, graphic, and vibration. When participants received skin stretch feedback, they localized the stiff region as well as with force feedback, with no increase in task completion time. Additionally, participants receiving skin-stretch feedback localized the stiff region statistically significantly more accurately than those using vibration feedback. Although participants using skin stretch exhibited higher interaction forces than when using force, vibration, and graphical feedback, skin stretch statistically significantly decreased interaction forces compared with reduced gain force feedback. Thus, skin-stretch feedback is a compelling substitute for force feedback and may be useful in scenarios where force feedback is reduced or infeasible.

Localizing tumors and measuring tissue mechanical properties can aid in surgical planning and evaluating the progression of disease. In this paper, autonomous robotic palpation with supervised machine learning algorithms enables mechanical localization and segmentation of stiff inclusions in artificial tissue. Elastography generates training data for the learning algorithms, providing a noninvasive, inclusion-specific characterization of tissue mechanics. Once an embedded hard inclusion was identified in the elastographic image, Gaussian discriminant analysis generated a classifier to threshold stiffness values acquired from autonomous robotic palpation. This classifier was later used to classify newly acquired points as either part of the inclusion or surrounding soft tissue. An expectation-maximization algorithm with underlying Markov random fields improved this initial classifier over successive iterations to better approximate the boundary of the inclusion. Results demonstrate robustness with respect to inclusion shape, size, and the initial classifier value. For three trials segmenting a cubic inclusion, sensitivity was above 0.95 and specificity was above 0.92.

Asymmetric-tip, robotically controlled steerable needles have the potential to improve clinical outcomes for many needle-based procedures by allowing the needle to curve and change direction within biological tissue. Algorithms have previously been developed to change the curvature of the needle trajectory via duty-cycled spinning. However, these algorithms require continuous rotation of the steerable needle, preventing the use of instrumentation such as force-torque sensors and electromagnetic trackers, due cable wind-up issues. In this paper, we present two novel control methods for duty-cycling a steerable needle without the need for continuous rotation: bidirectional duty-cycled spinning and duty-cycled flipping. These algorithms can be implemented on existing robotic needle steering systems without hardware changes. We evaluate our algorithms using a custom hollow steerable needle, with an embedded EM tracker and a force-torque sensor. We compare the path tracking error, needle insertion forces, and needle axial rotation torques of our algorithms and found no significant differences between the two algorithms in terms of tracking error. Duty-cycled flipping has significantly lower mean insertion forces and torques than bidirectional duty-cycled spinning, and differences in insertion force and rotation torque variability were also found. These results may have interesting implications for tissue health.

Telerobotics has the potential to facilitate the repair of satellites in geosynchronous orbit by allowing human operators to interact naturally with remote objects. Time delays on the order of seconds make it difficult to provide immersive feedback to the operator, motivating the use of predictive visual and haptic displays of the robot and environment. A teleoperation framework developed for this scenario invokes a two-part environment model that predicts motion of objects in the environment, both in free space and during contact with the robot. When objects in the environment are in free space, a propagated model using delayed data provides predictive feedback to the operator. However, when the robot interacts with the environment, a local environment model that does not propagate delayed data is used. This reduces computational load and ensures stability during robot-environment interactions. Two experiments were carried out to test the teleoperation system. Results demonstrate the ability of the prediction algorithm to provide reliable feedback and improve operator performance before, during, and after robot-environment interactions.

Robotic systems can improve percutaneous interventions by steering flexible needles along nonlinear trajectories. These systems require medical image feedback for accurate closed-loop control. Three-dimensional (3D) ultrasound can provide real-time measurements of needle pose within tissue; however, the ultrasound produces relatively large amounts of measurement noise. A recursive estimation approach is described for accurately estimating the six-degree-of-freedom pose of a steerable needle tip, by applying an unscented Kalman filter (UKF) to 3D ultrasound segmentation results. The UKF is formulated based on a kinematic process model of needle steering, as well as experimental quantification of the statistical variability of steering and imaging needles in biological tissue. Validation testing shows that the UKF method makes accurate closed-loop robotic control of the needle tip possible in biological tissue. Compared to direct use of noisy ultrasound data for control feedback, the UKF reduced average positioning error by 9.58 mm (81%) when steering towards a simulated target. This new estimation scheme will contribute towards the future evaluation of needle steering robots in real-world clinical applications.

During tool-mediated interactions with objects, we experience force and fingerpad skin stretch resulting from shear forces caused by friction between the fingerpad skin and the stylus. When probing an object, for the same penetration distance, a stiffer object causes a larger load force and, thus, greater fingerpad skin stretch. We hypothesized that rendering additional artificial skin stretch together with force will increase perceived stiffness. We created a Skin Stretch Stylus that renders skin stretch through tactor displacement, attached it to a force-feedback device, and performed a study to characterize the effect of tactor displacement-induced skin stretch on stiffness perception. Results showed that adding artificial skin stretch causes additive augmentation of perceived stiffness across a range of surface stiffness, and the addition is a linear function of tactor displacement gain. However, intersubject variability in the estimated slope coefficient was large. We propose a model that explains the additive effect and suggests potential sources for the intersubject variability. We conclude that augmenting force feedback with skin stretch can increase users' perception of stiffness, but the effect is user-specific. Such augmentation may be useful in virtual environment and teleoperation scenarios when force feedback gains must be kept low to prevent feedback-induced instabilities, or when force feedback is limited due to actuator force limits.

Robotically steered needles can improve clinical procedures by curving significantly within the body to attain targets and avoid obstacles. Needles that steer by tip asymmetry are nonholonomic systems, which are difficult to control manually (i.e. in joint space) due to under-actuation and unintuitive kinematic constraints. We propose a new teleoperation approach for nonholonomic systems (steerable needles in particular) that allows a user to command the desired position of a robot in Cartesian space and provides force feedback to represent kinematic constraints and the position error of the robot. We performed a user study with a virtual environment to evaluate the effectiveness of Cartesian space teleoperation in comparison to traditional joint space teleoperation, as well as the role of force feedback in Cartesian space teleoperation. Time-to-target and needle insertion length were significantly smaller for Cartesian space control than for joint space control, and when combined with force feedback, Cartesian space control resulted in significantly less targeting error than joint space control. Force feedback during Cartesian space control also reduced tracking error between the user and needle during insertion. Users rated Cartesian space control as easier overall; however, a few subjects felt they had less direct control of the needle.

When interacting with everyday objects, we experience kinesthetic force feedback as well as various forms of cutaneous tactile feed-back. Skin stretch is part of the cutaneous tactile experience that is caused by friction between the skin and the grasped object. Interacting with stiffer objects causes larger force, and results in a larger amount of skin stretch. Therefore, we hypothesize that adding artificial fingerpad skin stretch to kinesthetic force feedback will increase users' perception of stiffness. A tactile display called the Skin Stretch Stylus was designed to augment kinesthetic force feedback with skin stretch feedback. The change in users' stiffness perception due to the addition of skin stretch feedback is quantified through a two-alternative forced-choice paradigm, method of constant stimuli experiment. In this experiment, subjects compared the stiffness of virtual springs with kinesthetic force feedback augmented with skin stretch feedback versus virtual springs with only kinesthetic force feedback. Results show that the addition of skin stretch causes a significant increase in the perception of stiffness, and this effect increases with higher amount of applied skin stretch. These results indicate that skin stretch feedback could be used to augment perceived stiffness in situations where it is not possible to increase force feedback gains. Such scenarios include teleoperation systems where force feedback gains must remain low to ensure stability, and haptic devices with limited actuator force.

Many controllable tactile displays present the user with either variable mechanical properties or adjustable surface geometries, but controlling both simultaneously is challenging due to electromechanical complexity and the size/weight constraints of haptic applications. This paper discusses the design, manufacturing, control, and preliminary evaluation of a novel haptic display that achieves both variable stiffness and deformable geometry via air pressure and a technique called particle jamming. The surface of the device consists of a flat, deformable layer of hollow silicone cells filled with coffee grounds. It selectively solidifies in different regions when the air is vacuumed out of individual cells, jamming the coffee particles together. The silicone layer is clamped over a chamber with regulated air pressure. Different sequences of air pressure and vacuum level adjustment allow regions of the surface to display a small rigid lump, a large soft plane, and various other combinations of lump size and stiffness. Experimental data from individual cells show that surface stiffness increases with vacuum level and the elliptical shape of the cells become increasingly spherical with increased chamber pressure.

Localizing tumors and measuring tissue mechanical properties can be useful for surgical planning and evaluating progression of disease. In this paper, supervised machine learning algorithms enable mechanical localization of stiff inclusions in artificial tissue after autonomous robotic palpation. Elastography is used to generate training data for the learning algorithms, providing a non-invasive, inclusion-specific characterization of tissue biomechanics. In particular, elastography was used to characterize the stiffness of artificial tissue with an embedded hard inclusion. Once the inclusion was identified on the elastographic image, machine learning methods identified the difference in stiffness between the inclusion and surrounding soft tissue and generated classifiers, which were used to label stiffness values as either part of the inclusion or soft tissue. Next, data acquired via autonomous robotic palpation of the artificial tissue created a map of the stiffness distributed over the surface of the tissue. The points in this map were thresholded against the classifiers trained by the machine learning algorithms, and points theorized to belong to the hard inclusion were labeled. Centroid approximations of the hard inclusion based on this labeling show that classifying stiffness data acquired by autonomous robotic palpation and labeled by a classifier trained from elastography data provides a more accurate method of localizing hard inclusions than using unclassified data.

Teleoperated robot-assisted surgery (RAS) provides surgeons with improved dexterity, movement control, and visualization in comparison to standard minimally invasive surgery. However, there exists little quantitative understanding of the motor performance of human operators in RAS. Models of how users control their movements and how this control relates to surgical performance could provide inspiration for new robot or human interface designs, as well as more targeted training methods. Toward this end, we present a framework for the analysis of surgeon arm posture variability based on the uncontrolled manifold (UCM) concept, a method used in the study of human motor control for testing hypotheses about the coupling of control and task variables. We partition users' joint angle variability into variability that does and does not result in hand trajectory change. In a preliminary study applying this framework, we explored how expert and novice operators control planar reaching and reversal movements when moving freehand as well as using a teleoperated RAS system. We show that only movements in task-relevant directions are stabilized by the coordination of joint angles, and that this stabilization is stronger for expert movements than novice movements. We also show that stabilization is stronger in freehand than teleoperated movements, especially for the expert. These preliminary findings suggest that the proposed framework can be useful for: (1) assessment of teleoperator design and control that reveals how design parameters affect the ability of the user to exploit the UCM for stabilization of hand movement, and (2) skill assessment in RAS.

Skin stretch is a novel haptic feedback method that can provide a human operator with information about the magnitude and direction of an applied force. To evaluate the potential for skin stretch feedback to be used as a sensory substitute for kinesthetic (force) feedback in robotic teleoperation systems, a study was conducted to measure the ability of users to discriminate environment stiffness using varying levels of fingerpad skin stretch instead of force feedback. A new, high-fidelity skin stretch feedback device was developed that imposes tangential fingerpad skin stretch in proportion to the intended level of force feedback. In psychophysical experiments, users received skin stretch feedback with magnitude proportional to the users' penetration depth into a virtual wall. Users' stiffness discrimination capability using skin stretch was comparable to that of using force feedback. Furthermore, larger skin stretch cues were perceived by users as portraying greater stiffness without any advance training, which indicates that skin stretch feedback would be an intuitive sensory substitute for force feedback. Thus, skin stretch feedback is a promising method for conveying kinesthetic force information in applications such as robot-assisted surgery, where high levels of force feedback may not be desirable due to stability or safety concerns.

Needles provide percutaneous access to tissues deep within the body in order to deliver therapy or acquire biological samples for diagnostic purposes. The effectiveness of needle-based access often requires very accurate targeting of the needle tip within tissue. We designed a device that uses suction to fixate tissue during robotic needle insertion and measured its ability to improve targeting accuracy and decrease tissue deformation in artificial and ex vivo tissue. The deformation of the tissue was tracked in ultrasound images to evaluate the device's performance. The device was particularly effective at fixating ex vivo tissue with an attached skin layer and decreased tissue deformation by up to 96%. When accessing an embedded target within tissue, suction fixation reduced the targeting error by over 1 mm in chicken breast without a membrane and over 2 mm in chicken breast with an attached skin. Suction fixation could be used to improve the accuracy of robotic needle insertion through the skin and into internal organs with capsules.

Manipulating objects with an upper limb prosthesis requires significantly more visual attention than doing the same task with an intact limb. Prior work and comments from individuals lacking proprioception indicate that conveying prosthesis motion through a nonvisual sensory channel would reduce and possibly remove the need to watch the prosthesis. To motivate the design of suitable sensory substitution devices, this study investigates the difference between seeing a virtual prosthetic limb move and feeling one's real limb move. Fifteen intact subjects controlled a virtual prosthetic finger in a one-degree-of-freedom rotational spring discrimination task. A custom haptic device was used to measure both real finger position and applied finger force, and the resulting prosthetic finger movement was displayed visually (on a computer screen) and/or proprioceptively (by allowing the subject's real finger to move). Spring discrimination performance was tested for three experimental sensory conditions-visual motion, proprioceptive motion, and visual and proprioceptive motion-using the method of constant stimuli, with a reference stiffness of 290 N/m. During each trial, subjects sequentially pressed the right index finger on a pair of hard-surfaced virtual springs and decided which was stiffer. No significant performance differences were found between the three experimental sensory conditions, but subjects perceived proprioceptive motion to be significantly more useful than visual motion. These results imply that relaying proprioceptive information through a nonvisual channel could reduce visual attention during prosthesis control while maintaining task performance, thus improving the upper limb prosthesis experience.

Stable grasp is the result of sensorimotor regulation of forces, ensuring sufficient grip force and the integrity of the held object. Grasping with a prosthesis introduces the challenge of finding the appropriate forces given the engineered sensorimotor prosthetic interface. Excessive force leads to unnecessary energy use and possible damage to the object. In contrast, low grip forces lead to slippage. In order for a prosthetic hand to achieve a stable grasp, the haptic information provided to the prosthesis wearer needs to display these two antagonistic grasp metrics (force and slip) in a quantified way. We present the design and evaluation of a wearable single-actuator haptic device that relays multi-modal haptic information, such as grip force and slip speed. Two belts that are activated in a mutually exclusive manner by the rotation direction of a single motor exert normal force and tangential motion on the skin surface, respectively. The wearable haptic device is able to display normal forces as a tap frequency in the range of approximately 1.5–5.0 Hz and slip speed in the range of 50–200 mm/s. Within these values, users are able to identify at least four stimulation levels for each feedback modality, with short-term training.

Recent studies of neuromotor control have shown that humans modulate the impedance of their arms for different tasks, motivating the development of a prosthetic arm with user-selectable impedance characteristics. While impedance control has been extensively studied in robotics, prosthetic arms present a unique combination of human and robotic control, in which human capabilities and preferences play an important role. To understand the desirable impedance characteristics of prosthetic arms, we studied human control of a one-degree of-freedom virtual prosthetic limb with variable stiffness and damping in tasks involving (1) force minimization and (2) trajectory tracking. Subjects performed best with different patterns of impedance modulation depending on task goals and available feedback, suggesting that the ability to modulate the impedance of a prosthetic limb may be beneficial to the wearer. The results of this study inform the future design of prosthetic limbs in which the wearer can vary limb impedance to improve performance in a variety of manipulation tasks.

Detection of internal puncture during manual needle insertion is challenging due to the large friction force between a needle shaft and surrounding tissue, which masks small changes in force at the needle tip. A novel robotic coaxial needle insertion assistant was developed to enhance operator perception during epidural puncture. The coaxial needle separates the cutting force at the needle tip from shear friction on the needle shaft. When the assistant is inactive, the human operator controls the motion of the coaxial needle. When the assistant is active under position control, a motor controls the motion of the outer needle, while the operator controls only the motion of the inner needle. When the assistant is active under force control, the operator controls the motion of the entire needle, but receives force feedback that is a scaled version of the force applied by the needle tip to the tissue. The effectiveness of the assistant in enabling puncture detection was tested in an experiment in which users were asked to puncture artificial tissues with the assistant inactive and active. Results show that the ratio of successful to unsuccessful puncture detection was higher with the assistant than without. In addition, users were more confident that they could perceive the moment of puncture. However, when the users successfully perceived puncture, overshoot with the assistant was larger than without.

Despite the great diversity of teleoperator designs and applications, their underlying control systems have many similarities. These similarities can be exploited to enable inter-operability between heterogeneous systems. We have developed a network data specification, the Interoperable Telerobotics Protocol, that can be used for Internet based control of a wide range of teleoperators. In this work we test interoperable telerobotics on the global Internet, focusing on the telesurgery application domain. Fourteen globally dispersed telerobotic master and slave systems were connected in thirty trials in one twenty four hour period. Users performed common manipulation tasks to demonstrate effective master-slave operation. With twenty eight (93%) successful, unique connections the results show a high potential for standardizing telerobotic operation. Furthermore, new paradigms for telesurgical operation and training are presented, including a networked surgery trainer and upper-limb exoskeleton control of micro-manipulators.

Insertion velocity, tip asymmetry, and shaft diameter may influence steerable needle insertion paths in soft tissue. In this paper we examine the effects of these variables on needle paths in ex vivo goat liver, and demonstrate practical applications of robotic needle steering for ablation, biopsy, and brachytherapy. All experiments were performed using a new portable needle steering robot that steers asymmetric-tip needles under fluoroscopic imaging. For bevel-tip needles, we found that larger diameter needles resulted in less curvature, i.e. less steerability, confirming previous experiments in artificial tissue. The needles steered with radii of curvature ranging from 3:4 cm (for the most steerable pre-bent needle) to 2:97m (for the least steerable bevel needle). Pre-bend angle significantly affected needle curvature, but bevel angle did not. We hypothesize that biological tissue characteristics such as inhomogeneity and viscoelasticity significantly increase path variability. These results underscore the need for closed-loop image guidance for needle steering in biological tissues with complex internal structure.

Flexible needles with bevel tips are being developed as useful tools for minimally invasive surgery and percutaneous therapy. When such a needle is inserted into soft tissue, it bends due to the asymmetric geometry of the bevel tip. This insertion with bending is not completely repeatable. We characterize the deviations in needle tip pose (position and orientation) by performing repeated needle insertions into artificial tissue. The base of the needle is pushed at a constant speed without rotating, and the covariance of the distribution of the needle tip pose is computed from experimental data. We develop the closed-form equations to describe how the covariance varies with different model parameters. We estimate the model parameters by matching the closed-form covariance and the experimentally obtained covariance. In this work, we use a needle model modified from a previously developed model with two noise parameters. The modified needle model uses three noise parameters to better capture the stochastic behavior of the needle insertion. The modified needle model provides an improvement of the covariance error from 26.1% to 6.55%.

Proprioception plays a crucial role in enabling humans to move purposively and interact with their physical surroundings. Current technology in upper-limb prostheses, while beginning to incorporate some haptic feedback, does not provide amputees with proprioceptive information about the state of the limb. Thus, the wearer must visually monitor the limb, which is often inconvenient or even impossible for some tasks. This work seeks to quantify the potential benefits of incorporating proprioceptive motion feedback into upper-limb prosthesis designs. We apply a noninvasive method for controlling the availability of proprioceptive motion feedback in unimpaired individuals in a human subject study to compare the benefits of visual and proprioceptive motion feedback in targeted motion tasks. Combined results of the current study and our previous study using a different task indicate that the addition of proprioceptive motion feedback improves targeting accuracy under nonsighted conditions and, for some tasks, under sighted conditions as well. This work motivates the development of methods for providing artificial proprioceptive feedback to a prosthesis wearer.

Haptic virtual fixtures have been shown to improve user performance and increase the safety of robot-assisted tasks, particularly for surgical applications. However, little research has studied virtual fixtures that provide moving force constraints based on motion of the environment, e.g., organ movement due to heartbeat or respiration. This work discusses design considerations of moving forbidden-region virtual fixtures and presents two methods of implementation: predicted-position and current-position virtual fixtures. Human subject experiments were performed to determine the effectiveness of moving virtual fixtures when interacting with an object in motion using a teleoperator. Results show that moving virtual fixtures can help improve user precision and decrease the amount of force applied.

Direct haptic feedback and graphical force feedback have both been hypothesized to improve the performance of robot-assisted surgery. In this study we evaluate the benefits of haptic and graphical force feedback on surgeon performance and tissue exploration behavior during a teleoperated palpation task of artificial tissues. Seven surgeon subjects (four experienced in robot-assisted surgery) used a 7-degree-of-freedom teleoperated surgical robot to identify a comparatively rigid rigid target object (representing a calcified artery) in phantom heart models using the following feedback conditions: (1) direct haptic and graphical feedback, (2) direct haptic only, (3) graphical feedback only, and (4) no feedback. To avoid the problems of force sensing in a minimally invasive surgical environment, we use a position-exchange controller with dynamics compensation for direct haptic feedback and a force estimator displayed via tool-tip tracking bar graph for graphical force feedback. Although the transparency of the system is limited with this approach, results show that direct haptic force feedback minimizes applied forces to the tissue, while coupled haptic and graphical force feedback minimizes subject task error. For experienced surgeons, haptic force feedback substantially reduced task error independent of graphical feedback.

The asymmetry of a bevel-tip needle results in the needle naturally bending when it is inserted into soft tissue. In this study we present a mechanics-based model that calculates the deflection of the needle embedded in an elastic medium. Microscopic observations for several needle-gel interactions were used to characterize the interactions at the bevel tip and along the needle shaft. The model design was guided by microscopic observations of several needle-gel interactions. The energy-based model formulation incorporates tissue-specific parameters such as rupture toughness, nonlinear material elasticity, and interaction stiffness, and needle geometric and material properties. Simulation results follow similar trends (deflection and radius of curvature) to those observed in macroscopic experimental studies of a robot-driven needle interacting with different kinds of gels. These results contribute to a mechanics-based model of robotic needle steering, extending previous work on kinematic models.

A flexible needle can be accurately steered by robotically controlling the orientation of the bevel tip as the needle is inserted into tissue. Here, we demonstrate the significant effect of friction between the long, flexible needle shaft and the tissue, which can cause a significant discrepancy between the orientation of the needle tip and the orientation of the base where the needle is controlled. Our experiments show that several common phantom tissues used in needle steering experiments impart substantial frictional forces to the needle shaft, resulting in a lag of over 45° for a 10 cm insertion depth in some phantoms; clinical studies have reported torques large enough to could cause similar errors during needle insertions. Such angle discrepancies will result in poor performance or failure of path planners and image-guided controllers, since the needles used in percutaneous procedures are too small for state-of-the-art imaging to accurately measure the tip angle. To compensate for the angle discrepancy, we develop a model for the rotational dynamics of a needle being continuously inserted into tissue and show how a PD controller is sufficient to compensate for the rotational dynamics.

Manual palpation of tissue and organs during a surgical procedure provides clinicians with valuable information for diagnosis and surgical planning. In present-day robotassisted minimally invasive surgery systems, lack of perceptible haptic feedback makes it challenging to detect a tumor in an organ or a calcified artery in heart tissue. This study presents an automated tissue property estimation method and a real-time graphical overlay that allow an operator to discriminate hard and soft tissues. We first evaluate experimentally the properties of an artificial tissue and compare seven possible mathematical tissue models. Self-validation as well as cross-validation confirm that the Hunt-Crossley model best describes the experimentally observed phantom tissue properties and is suitable for our purpose. Second, we present the development of a system in which the phantom tissue is palpated using a teleoperated surgical robot, and the stiffness of the Hunt-Crossly model is estimated in real time by recursive least squares. A real-time visual overlay representing tissue stiffness is created using a hue-saturation-luminance representation on a semi-transparent disc at the tissue surface. Hue depicts the stiffness at a palpated point and saturation is calculated based on distance from the point. A simple interpolation technique creates a continuous stiffness color map. In an experiment, the graphical overlay successfully shows the location of an artificial calcified artery hidden in phantom tissue.

This work focuses on the implementation of a vision-based motion guidance method, called virtual fixtures, on admittance-controlled human-machine cooperative robots with compliance. The robot compliance here refers to the structural elastic deformation of the device. The high mechanical stiffness and non-backdrivability of a typical admittance-controlled robot allow for slow and precise motions, making it highly suitable for tasks that require accuracy near human physical limits, such as microsurgery. However, previous experiments have shown that even small robot compliance degraded virtual fixture performance, especially at the micro scale. In this work, control methods to minimize the effect of robot compliance on virtual fixture performance were developed for admittance-controlled cooperative systems. Based on a linear model of the robot dynamics, we applied a Kalman filter to integrate the measurements obtained from the camera and encoders to estimate the robot end-effector position. A partitioned control law was used to achieve end-effector trajectory following on the desired velocity commanded by the admittance and virtual fixture control laws. The effectiveness of the Kalman filter and the controller was validated on a one degree-of-freedom admittance-controlled cooperative testbed.

In human-machine collaborative systems, robot joint compliance and human-input dynamics lead to involuntary tool motion into undesired regions. To correct this, a set of methods, called dynamically-defined virtual fixtures, was previously proposed to create a movable virtual fixture that stops the user at a safe distance outside the forbidden region. In this work, a new method, called the force-based method, was added. A vision system was introduced for real-time tool tracking. Additionally, we implemented a closed-loop controller with the virtual fixtures that allows the user to reach, but not enter, the forbidden region. Two user experiments were conducted on a 1-DOF testbed to evaluate the virtual fixture methods. The first experiment showed the effectiveness of the virtual fixtures in preventing the penetration. However, the absence of haptic feedback in the closed-loop implementation resulted in boundary penetration. In the second experiment, visual feedback was used to compensate for the lack of haptic feedback. User cognitive load was added as an inhibiting factor in a human-machine cooperative setting. The experiment showed a significant reduction in penetration with visual feedback, while the addition of cognitive load did not significantly increase the penetration.

Human machine collaborative systems (HMCS) have been developed to enhance sensation and suppress extraneous motions or forces during surgical tasks requiring precise motion. However, to date such systems have enforced constraints on the position or path of a tool, but have not considered the dynamics of motion. Also, the focus has been on the effect of guidance of motion during a task, rather than on the learning of motion skills through repetition. We present a pseudo-admittance framework for HMCS design to guide the user's velocity in such tasks. Two different fixture design approaches are analyzed, implemented and compared. Three tests are then conducted, showing the fixtures' promise for both guiding and learning motions with dynamics

Needles are commonly used in medical practice as a minimally invasive means to reach subsurface targets for diagnosis or therapy delivery. Recent results indicate that steerable needles may enhance targeting accuracy and allow needles to avoid obstacles along the path to the target. This work considers teleoperation of needles made of a superelastic alloy that steer through tissue using forces generated by the standard asymmetric bevel tip. The needle may be modeled as a nonholonomic system, with inputs of insertion along and spin about the needle axis. A teleoperation system consisting of a commercial master haptic device, a custom needle-steering robot slave, and visual feedback to the operator was assembled. Human subjects experiments were performed to evaluate targeting accuracy in phantom tissue for three needle control methods: teleoperation of both insertion and spin, teleoperation of insertion with open-loop-controlled spin, and open-loop control of both insertion and spin. Targeting accuracy improved with increasing degrees of freedom of human (teleoperation) control, primarily because tissue deformation and modeling limitations result in open-loop control errors. Subjects typically performed multiple spins of the needle during insertion in order to fine tune the needle path. In addition, position, rate, and a nonlinear hybrid control were compared during teleoperation of the insertion degree of freedom. The hybrid method resulted in significantly better targeting accuracy

This article presents a model-based compensator for canceling friction in the tendon-driven joints of a haptic-feedback teleoperator. Unlike position-tracking systems, a teleoperator involves an unknown environment force that prevents the use of tracking position error as a feedback to the compensator. Thus, we use a model-based feedforward friction compensator to cancel the friction forces. We provide conditions for selecting compensator parameters to ensure passivity of the teleoperator and demonstrate performance experimentally.

Virtual fixtures are virtual constraints that enhance human performance in motion tasks. They can either confine and/or guide a user's motion. In this paper, we use a commercially available motion platform to explore the portability and applicability of virtual fixtures and document how people interact with them. Two micromanipulation tasks are analyzed and the effects of similarly designed virtual fixtures are discussed. One task simulates a medical task, retinal vein cannulation, and the other simulates a manufacturing task, fine leads soldering. Preliminary experimental results show that the virtual fixtures increase the accuracy of both medical and manufacturing tasks, lending support to its portability and applicability across unrelated tasks

This paper presents a model-based approach to cancel friction in the joints of the manipulators of a force-feedback telerobotic system. Friction compensation can improve the transparency of telerobotic systems, where transparency is quantified in terms of a match between the impedance of the environment and the impedance transmitted to the user. We used Dahl friction models to compensate for physical friction in the device. Experiments performed on a telerobotic system demonstrated that teleoperation transparency is improved by using these models. Further, the stability of the teleoperation is analyzed using passivity theory, and it is shown that the master-slave system remains stable up to a certain level of friction compensation

We have developed a new class of continuously flexible snake-like robots, called active cannulas, that consist of several telescoping pre-curved superelastic tubes. The devices derive bending actuation not from tendon wires or other external mechanisms, but from elastic energy stored in the backbone itself. This allows active cannulas to have a small diameter and a high degree of dexterity, which should enable them to navigate through complex anatomy to sites inaccessible by current surgical robotic devices. Active cannulas may also enhance patient safety because their inherent compliance mitigates potential trauma from inadvertent tool-tissue collision. A consequence of our design is that dexterity improves with miniaturization. A kinematic description of active cannula shape requires a model of the elastic interaction of telescoping pre-curved flexible tubes, and we derive a two-"link" beam mechanics-based model. Experiments using curved nitinol tubes and wires validate the model.

We explore motion planning for a new class of highly flexible bevel-tip medical needles that can be steered to previously unreachable targets in soft tissue. Planning for these procedures is difficult because the needles bend during insertion and cause the surrounding soft tissues to displace and deform. In this paper, we develop a planning algorithm for insertion of highly flexible bevel-tip needles into soft tissues with obstacles in a 2D imaging plane. Given an initial needle insertion plan specifying location, orientation, bevel rotation, and insertion distance, the planner combines soft tissue modeling and numerical optimization to generate a needle insertion plan that compensates for simulated tissue de formations, locally avoids polygonal obstacles, and minimizes needle insertion distance. The simulator computes soft tissue deformations using a finite element model that incorporates the effects of needle tip and frictional forces using a 2D mesh. We formulate the planning problem as a constrained nonlinear optimization problem that is locally minimized using a penalty method that converts the formulation to a sequence of unconstrained optimization problems. We apply the planner to bevel-right and bevel-left needles and generate plans for targets that are unreachable by rigid needles.

Fine needles facilitate diagnosis and therapy because they enable minimally invasive surgical interventions. This paper formulates the problem of steering a very flexible needle through firm tissue as a nonholonomic kinematics problem, and demonstrates how planning can be accomplished using diffusion-based motion planning on the Euclidean group, SE(3). In the present formulation, the tissue is treated as isotropic and no obstacles are present. The bevel tip of the needle is treated as a nonholonomic constraint that can be viewed as a 3D extension of the standard kinematic cart or unicycle. A deterministic model is used as the starting point, and reachability criteria are established. A stochastic differential equation and its corresponding Fokker-Planck equation are derived. The Euler-Maruyama method is used to generate the ensemble of reachable states of the needle tip. Inverse kinematics methods developed previously for hyper-redundant and binary manipulators that use this probability density information are applied to generate needle tip paths that reach the desired targets.

Many medical procedures involve the use of needles, but targeting accuracy can be limited due to obstacles in the needle’s path, shifts in target position caused by tissue deformation, and undesired bending of the needle after insertion. In order to address these limitations, we have developed robotic systems that actively steer a needle in soft tissue. A bevel (asymmetric) tip causes the needle to bend during insertion, and steering is enhanced when the needle is very flexible. An experimental needle steering robot was designed that includes force/torque sensing, horizontal needle insertion, stereo image data acquisition, and controlled actuation of needle rotation and translation. Experiments were performed with a phantom tissue to determine the effects of insertion velocity and bevel tip angle on the needle path, as well as the forces acting on the needle during insertion. Results indicate that needle steering inside tissue does not depend on insertion velocity, but does depend on bevel tip angle. In addition, the forces acting on the needle are directly related to the insertion velocity.

When inserted into soft tissues, flexible needles with bevel tips have been shown experimentally to follow a path of constant curvature in the direction of the bevel. By controlling 2 degrees of freedom at the needle base (bevel direction and insertion distance), these needles can be steered around obstacles to reach targets inaccessible to rigid needles. Motion planning for needle steering is a type of nonholonomic planning for a Dubins car with no reversal. We develop a motion planning algorithm based on dynamic programming where the path of the needle is uncertain due to uncertainty in tissue properties, needle mechanics, and interaction forces. The algorithm computes a discrete control sequence of insertions and direction changes so the needle reaches a target in an imaging plane while minimizing expected cost due to insertion distance, direction changes, and obstacle collisions. We efficiently sample the state space of needle tip positions and orientations and define bounds on the errors due to discretization. We formulate the motion planning problem as a Markov decision process (MDP) and use infinite horizon dynamic programming to compute an optimal control sequence. We first apply the method to the deterministic motion case where the needle precisely follows a path of constant curvature and then to the uncertain motion case where state transitions are defined by a probability distribution. Our implementation generates motion plans for bevel-tip needles that reach targets inaccessible to rigid needles and demonstrates that accounting for uncertainty can lead to significantly different motion plans.

A novel two-degree-of-freedom tactile display reproduces the sensations of sliding contact and incipient slip through the rotation of a ball positioned under the user's fingertip. A pair of motor-driven wheels actuates the ball via contact friction. Mechanical performance requirements are used to define the dimensions and construction method of the device. Kinematic analysis shows that the drive wheel angles and their contact locations with the ball must be carefully selected in order to accurately control the axis of rotation and speed of the ball. However, psychophysical experiments indicate that some kinematic error is tolerable; errors of up to 20° in slip angle and 30% of a nominal velocity may be applied without detection from an average user. The lightweight, modular tactile display was attached to a multi-degree-of-freedom kinesthetic interface and used to display virtual environments with slip. Experimental results demonstrate that users complete a virtual paper manipulation task with lower applied forces using combined slip and force feedback in comparison with conventional force feedback alone.

The "virtual wall" is the most common building block used in constructing haptic virtual environments. A virtual wall is typically based on a simple spring model, with unilateral constraints that allow the user to make and break contact with a surface. There are a number of factors (sample-and-hold, device dynamics, sensor quantization, etc.) that cause virtual walls to demonstrate active (nonpassive) behavior, destroying the illusion of reality. In this paper, we find an explicit upper bound on virtual wall stiffness that is a necessary and sufficient condition for virtual wall passivity. We consider a haptic display that can be modeled as a mass with Coulomb-plus-viscous friction, being acted upon by two external forces: an actuator and a human user. The system is equipped with only one sensor, an optical encoder measuring the position of the mass. We explicitly model the effects of position resolution, which has not been done in previous work. We make no assumptions about the human user, and we consider arbitrary constant sampling rates. The main result of our analysis is a necessary and sufficient condition for passivity that relies on the Coulomb friction in the haptic device, as well as the encoder resolution. We experimentally verify our results with a one-degree-of-freedom haptic display, and find that the system can display nonpassive behavior in two decoupled modes that are predicted by the necessary and sufficient condition. One mode represents instability, while the other mode results in active tactile sensations.

We have designed a minimally invasive medical device with the capability to insert a needle (or catheter) under transrectal ultrasound (TRUS) image guidance through the rectum and into the surrounding tissue, primarily the prostate. A partial sheath surrounds an ultrasound probe 210/spl deg/ around the circumference in order to remain attached to the probe but minimize interference with the ultrasound image. A needle is inserted through one of two parametric guides on the sheath; which guide depends upon the desired location of the needle and the presence of anatomy to be avoided. Our hypothesis is that by reducing the amount of tissue, muscle, and nerves in the path of the needle, this method, when compared to the perineal approach, improves both the accuracy of the needle placement and the comfort to the patient. Experiments with a custom phantom prostate demonstrate that this device provides target accuracy comparable to the current perineal approach.

A real-time evolving neural network controller (RTENNC) for controlling teleoperation systems with uncertainties is presented. Two different slave position controllers, the RTENNC and a proportional-integral-derivative (PID) controller, were experimentally compared using it one-degree-of-freedom test bed. With both controllers, force feedback to the master is obtained using measurements from a force sensor mounted on the slave manipulator. When various environments are explored with this system, the RTENNC provides better impedance reflection than the PID controller. For each environment, position tracking, force reflection, and the impedance ratio (the ratio of the sensed environment impedance to the impedance displayed to the human operator) were recorded in order to determine a measure of system transparency. "Adaptive transparency," the ability of simultaneous position following and force tracking between the master and slave for changing environment impedances, was higher for the RTENNC. The RTENNC resulted in an impedance ratio of approximately 1.0, whereas the PID controller had errors of about 20%.

We present the design and implementation of a vision-based system for cooperative manipulation at millimeter to micrometer scales. The system is based on an admittance control algorithm that implements a broad class of guidance modes called virtual fixtures. A virtual fixture, like a real fixture, limits the motion of a tool to a prescribed class or range of motions. We describe how both hard (unyielding) and soft (yielding) virtual fixtures can be implemented in this control framework. We then detail the construction of virtual fixtures for point positioning and curve following as well as extensions of these to tubes, cones, and sequences thereof. We also describe an implemented system using the JHU Steady Hand Robot. The system uses computer vision as a sensor for providing a reference trajectory, and the virtual fixture control algorithm then provides haptic feedback to implemented direct, shared manipulation. We provide extensive experimental results detailing both system performance and the effects of virtual fixtures on human speed and accuracy.

A forbidden-region virtual fixture (FRVF) is a computer-generated constraint that displays position or force limitations to a robot manipulator or operator, in order to prevent motion into forbidden regions of the workspace. We compare nine FRVFs on each of four common telemanipulator control architectures: position forward, position exchange, position forward/force feedback, and position exchange/force feedback. A one-degree-of-freedom telemanipulation system was used in an experiment designed to simulate users working near a known forbidden region. The metrics of tracking, safety, and submittance were used to analyze the performance of the system with six different users. The results indicate that different FRVF architectures perform best for each of the three metrics. No single FRVF scheme is the best over all metrics, so selection of an FRVF architecture should be an application-dependent weighting of the three metrics. Across all control architectures, the results indicate that a very strong FRVF at the slave device in combination with no FRVF at the master device leads to poor telepresence.

In this article, we describe and demonstrate control algorithms for general motion constraints. These constraints are designed to enhance the accuracy and speed of a user manipulating in an environment with the assistance of a cooperative or telerobotic system. Our method uses a basis of preferred directions, created off-line or in real-time using sensor data, to generate virtual fixtures that may constrain the user to a curve, surface, orientation, etc. in space. Open loop virtual fixtures seek only to maintain user motion along preferred directions, whereas closed loop fixtures additionally guide the user toward a point, line, or surface. This article demonstrates and compares the effects of open and closed loop fixtures in both autonomous and human-machine cases.

Two experiments were performed to determine the effects of friction and needle geometry during robotic needle insertion into soft tissues. In Experiment I, friction forces along the instrument axis were characterized during needle insertion into bovine liver under CT fluoroscopic imaging. Because the relative velocity of the tissue and needle affect viscous and Coulomb friction, the needle insertion process was segmented into several phases of relative motion: none, partial and complete. During the complete relative motion phase, it was found that Coulomb friction accounts for the majority of needle force. In Experiment II, insertion forces along and orthogonal to the needle axis were measured during insertion into a silicone rubber phantom with a consistency similar to liver. The effects of needle diameter and tip type (bevel, cone, and triangle) on insertion force were characterized. A bevel tip causes more needle bending and is more easily affected by tissue density variations. Forces for larger diameter needles are higher due to increased cutting and friction forces. These results may be used in the control of needle insertion for robot-assisted percutaneous therapies.

The "haptic scissors" is a device that creates the sensation of cutting in virtual environments. The scissors have two degrees of freedom of motion and force feedback, one for cutting (single blade rotation) and one for translation. An algorithm was developed to simultaneously display translational and cutting forces for a realistic cutting simulation. In previous work, we use filtered data from cutting biological tissues to create "haptic recordings" of the cutting experience. Here, we consider two cutting models: one based on real tissue data and one that is analytical. The model based on real tissue is a segmented linear empirical model of the original data. Experimental results show that users cannot differentiate between these models and the haptic recordings created earlier. The analytical model uses a combination of friction, assumed material properties, and user motion (position and velocity) to determine the displayed cutting forces.

A forbidden-region virtual fixture (FRVF) is a constraint, implemented in software, that keeps the slave manipulator of a master/slave telemanipulation system from entering into a forbidden region of the workspace. In this paper, we consider the problem of unstable vibrations of the slave against the FRVF for a general class of telemanipulator control architectures. The master and slave equilibrium positions resulting from a constant human input force are found, and the system is evaluated around this equilibrium. We consider two methods of analyzing the stability of the system around this equilibrium point. The first method uses tools developed for analysis of two-port networks. The second method converts the system to its discrete state-space form, and then uses the position of the eigenvalues to analyze system stability. We find that the discrete state-space method agrees with simulations, and can be easily used to design and analyze the stability and transient behavior of the telemanipulator.

Surgical robots, including those with remote center of motion (RCM) mechanisms, have demonstrated utility in image-guided percutaneous needle placement procedures. However, widespread clinical application of these robots is hindered by not only complicated mechanical design but also the need for calibration and registration of the robot to the medical imager prior to each use. In response, we propose a Virtual RCM algorithm that requires only online tracking or registering the surgical tool to the imager, and a five degree-of-freedom (DOF) robot comprised of three prismatic DOF decoupled from two rotational DOF. The robot can be unencoded, uncalibrated, and does not require pre-operative registration. An incremental adaptive motion control cycle both guides the needle to the insertion point and orients it to align with the target. The robot executes RCM motion “virtually” without having a physically constrained fulcrum point. The proof-of-concept prototype system achieved 0.78 mm translation and 1.4 degrees rotational accuracy (within the tracker accuracy), within 17 iterative steps (0.5–1s).

This paper presents a set of algorithms for use in simultaneous localization and mapping during haptic exploration. Several solutions are provided for the problem of a single spherical robot finger exploring a known smooth surface, starting with an unknown pose. Using pose estimates, pattern matching is performed between the robot's internal model of the surface and the known model. The robot finger is guided to explore regions of the surface that will maximize the probability of recognition. Simulation results demonstrate the effectiveness of one algorithm. In addition, it is shown that haptic exploration and dexterous manipulation can be achieved concurrently when multiple robot fingers are used.

Haptic devices have been successfully incorporated into the human-computer interaction model. However, a drawback common to almost all haptic systems is that the user must be attached to the haptic devices at all times even though force feedback is not always being rendered. This constant contact hinders perception of the virtual environment, primarily because it prevents the user from feeling new tactile sensations upon contact with virtual objects. We present the design and implementation of an augmented reality system called VisHap that uses visual tracking to seamlessly integrate force feedback with tactile feedback to generate a "complete" haptic experience. The VisHap framework allows the user to interact with combinations of virtual and real objects naturally, thereby combining active and passive haptics. An example application of this framework is also presented. The flexibility and extensibility of our framework is promising in that it supports many interaction modes and allows further integration with other augmented reality system.

We present the design and implementation of a vision-based system for micron-scale, cooperative manipulation of a surgical tool. The system is based on a control algorithm that implements a broad class of guidance modes called virtual fixtures. A virtual fixture, like a real fixture, limits the motion of a tool to a prescribed class or range. The implemented system uses vision as a sensor for providing a reference trajectory, and the control algorithm then provides haptic feedback involving direct, shared manipulation of a surgical tool. We have tested this system on the JHU Steady Hand robot and provide experimental results for path following and positioning on structures at both macroscopic and microscopic scales.

Force information from needle insertions was measured and modeled for use in robot-assisted percutaneous therapies. Data was collected on bovine livers using the Johns Hopkins University Steady Hand Robot, and modeled in three parts: force due to capsule stiffness, friction, and cutting. Capsule stiffness is modeled by a nonlinear spring model, friction by a modified Karnopp model, and cutting by the remaining forces, which appear to be constant for a given tissue sample. During robot-assisted procedures, real-time force data can be compared to these models to control puncture of interior structures.

In this paper, we present experiments to determine the effect of different virtual fixture compliance levels in a human-machine cooperative manipulation system. Subjects used the JHU Steady Hand Robot with vision-based virtual fixtures to perform three common tasks: curve following, off-path targeting, and object avoidance. The virtual fixtures provided different levels of guidance to the operator, ranging from no guidance to complete guidance. User performance was evaluated based on the error and time for task execution. We developed an algorithm to determine the appropriate compliance level based on the nature of the task. Task parameters considered were time vs. accuracy and user constraint vs. freedom.

Haptic exploration with robotic fingers is accomplished by feature-guided exploration, where information about surface features such as cracks and ridges is used to guide the finger in an exploratory procedure. A local exploration strategy uses contact trajectory information from a tactile sensor to identify features while moving over and around them. Using an algorithm based on the Voronoi diagram, an approximation of the medial axis of the feature is found, then pruned using an edge length threshold. Multiple feature skeletons are then used to create a global skeleton that partitions the surface into regions. The models resulting from these local and global explorations can be used to characterize objects for information storage and manipulation planning.

The "steady hand" concept is a way of providing assistance for direct manipulation by applying constraints on the motion of a tool shared by a user and a robot. We explore in detail one family of constraints: virtual fixtures for use in path following tasks. Vision is used to sense the desired path, and then the robot encourages motion toward and along the path through a direction-based control law. This "soft" virtual fixture allows the user to move in other, non-preferred directions, maintaining the user's sense of autonomy and control. Experimental results show that user performance in assisted path following improves with virtual fixture augmentation, and differs with varying fixture compliance.

Presents an overview of research in dexterous manipulation. We first define robotic dexterous manipulation in comparison to traditional robotics and human manipulation. Next, kinematics, contact types and forces are used to formulate the dexterous manipulation problem. Dexterous motion planning is described, which includes grasp planning and quality measures. We look at mid- and low-level control frameworks, and then compare manipulation versus exploration. Finally, we list accomplishments in the different areas of dexterous manipulation research, and highlight important areas for future work.

We consider the detection of small surface features, such as ridges and bumps, on the surface of an object during dextrous manipulation. First, we review the representation of object surface geometry and present definitions of surface features based on local curvature. These definitions depend on the geometries of both the robot fingertips and the object being explored. We also show that the trajectory traced by a round fingertip rolling or sliding over the object surface has some intrinsic properties that facilitate feature detection. Next, several algorithms based on the feature definitions are presented and compared. Finally, we present simulated and experimental results for feature detection using a hemispherical fingertip equipped with an optical tactile sensor.

Vibrations can significantly enhance touch perception for virtual environment applications with minimal design complexity and cost. In order to create realistic vibrotactile feedback, we collected vibrations, forces, and velocities during various tasks executed with a stylus: tapping on materials, stroking textures, and puncturing membranes. Empirical models were fit to these waveforms and a library of model parameters was compiled. These models simulated tasks involving simultaneous display of forces and vibrations on a high-bandwidth force-feedback joystick. Vibration feedback adds little complexity to virtual environment algorithms. Human subjects interacting with the system showed improved execution and perception when performing surface feature discrimination tasks.

We present an approach for haptic exploration of unknown objects with dextrous robotic hands. The emphasis is on developing a robust manipulation process that allows fingers to traverse the surface of an object. The process consists of a sequence of phases in which some fingers are responsible for grasping and manipulating the object while others roll and slide over the object surface. The rolling/sliding fingers can utilize sensors to determine surface properties such as texture, friction or small features such as grooves and ridges. Simulations and experiments with a two-fingered hand were conducted to investigate the robustness of the approach for exploring various object shapes.