Electrosurgery is a surgical technique that can improve tissue cutting by reducing cutting force and bleeding. However, electrosurgery adds a risk of thermal injury to surrounding tissue. Expert surgeons estimate desirable cutting velocities based on experience but have no quantifiable reference to indicate if a particular velocity is optimal. Furthermore, prior demonstrations of autonomous electrosurgery have primarily used constant tool velocity, which is not robust to changes in electrosurgical tissue characteristics, power settings, or tool type. Thermal imaging feedback provides information that can be used to reduce thermal injury while balancing cutting force by controlling tool velocity. We introduce Thermography for Electrosurgical Rate Modulation via Optimization (ThERMO) to autonomously reduce thermal injury while balancing cutting force by intelligently controlling tool velocity. We demonstrate ThERMO in tissue phantoms and compare its performance to the constant velocity approach. Overall, ThERMO improves cut success rate by a factor of three and can reduce peak cutting force by a factor of two. ThERMO responds to varying environmental disturbances, reduces damage to tissue, and completes cutting tasks that would otherwise result in catastrophic failure for the constant velocity approach.

Humans possess an innate ability to incorporate tools into our body schema to perform a myriad of tasks not possible with our natural limbs. Human-in-the-loop telerobotic systems (HiLTS) are tools that extend human manipulation capabilities to remote and virtual environments. Unlike most hand-held tools, however, HiLTS often possess complex electromechanical architectures that introduce non-trivial transmission dynamics between the robot's leader and follower, which alter or obfuscate the environment's dynamics. While considerable research has focused on negating or circumventing these dynamics, it is not well understood how capable human operators are at incorporating these transmission dynamics into their sensorimotor control scheme. To begin answering this question, we recruited$\mathrm{N}=12$participants to use a novel reconfigurable teleoperator with varying transmission dynamics to perform a visuo-haptic tracking task. Contrary to our original hypothesis, our findings demonstrate that humans can account for substantial differences in teleoperator transmission dynamics and produce the compensatory strategies necessary to adequately control the teleoperator. These findings suggest that advances in transparency algorithms and haptic feedback approaches must be coupled with control designs that leverage the unique capabilities of the human operator in the loop.

Human-in-the-loop telerobotic systems (HiLTS) are robotic tools designed to extend and in some circumstances improve the dexterous capabilities of the human operator in virtual and remote environments. Dexterous manipulation, however, depends on how well the telerobot is incorporated into the operator’s sensorimotor control scheme. Empirical evidence suggests that haptic feedback can lead to improved dexterity. Unfortunately, haptic feedback can also introduce dynamics between the leader and follower of the telerobot that affect both stability and device performance. While concerted research effort has focused on masking these device dynamics or bypassing them altogether, it is not well understood how human operators incorporate these dynamics into their control scheme. We believe that to advance dexterous telerobotic manipulation, it is crucial to understand the process by which human operators incorporate teleoperator dynamics and distinguish them from the dynamics of the environment. Key to this knowledge is an understanding of how advanced telerobotic architectures compare to the gold standard, the rigid mechanical teleoperators first introduced in the 1950’s. In this manuscript, we present a teleoperator testbed that has reconfigurable transmissions between the leader and follower to change its dynamic behavior. The intent of this testbed is to investigate the effect of the teleoperator’s dynamics on perception of and task performance in the remote/virtual environment. We describe the hardware and software components of the testbed and then demonstrate how the different teleoperator transmissions can lead to differences, sometimes significant, in the dynamics that would be felt by the operator when exploring the same environment.

The lack of haptically aware upper-limb prostheses forces amputees to rely largely on visual cues to complete activities of daily living. In contrast, non-amputees inherently rely on conscious haptic perception and automatic tactile reflexes to govern volitional actions in situations that do not allow for constant visual attention. We therefore propose a myoelectric prosthesis system that reflects these concepts to aid manipulation performance without direct vision. To implement this design, we constructed two fabric-based tactile sensors that measure contact location along the palmar and dorsal sides of the prosthetic fingers and grasp pressure at the tip of the prosthetic thumb. Inspired by the natural sensorimotor system, we use the measurements from these sensors to provide vibrotactile feedback of contact location and implement a tactile grasp controller with reflexes that prevent over-grasping and object slip. We compare this tactile system to a standard myoelectric prosthesis in a challenging reach-to-pick-and-place task conducted without direct vision; 17 non-amputee adults took part in this single-session between-subjects study. Participants in the tactile group achieved more consistent high performance compared to participants in the standard group. These results show that adding contact-location feedback and reflex control increases the consistency with which objects can be grasped and moved without direct vision in upper-limb prosthetics.

Evaluations of haptic feedback in myoelectric prostheses are generally limited to task performance outcomes, which while necessary, fail to capture the mental effort of the user operating the prosthesis. Cognitive load is usually investigated with reaction time metrics and secondary task accuracy, which are indirect, and may not capture the time-varying nature of mental effort. Here, we propose wearable, wireless functional near infrared spectroscopy (fNIRS) neuroimaging to provide a continuous direct assessment of operator mental effort during use of a prosthesis. Utilizing fNIRS in a two-alternative forced-choice stiffness discrimination task, we asked participants to differentiate objects using their natural hand, a (traditional) myoelectric prosthesis without sensory feedback, and a myoelectric prosthesis with haptic (vibrotactile) feedback of grip force. Results showed that discrimination accuracy and mental effort are optimal with the natural hand, followed by the prosthesis featuring haptic feedback, and then the traditional prosthesis, particularly for objects whose stiffness were difficult to differentiate. This experiment highlights the utility of haptic feedback in improving task performance and lowering cognitive load for prosthesis use, and demonstrates the potential for fNIRS to provide a robust measure of cognitive effort for other human-in-the-loop systems.

Achieving dexterous volitional control of an upper-limb prosthetic device will require multimodal sensory feedback that goes beyond vision. Haptic display is well-positioned to provide this additional sensory information. Haptic display, however, includes a diverse set of modalities that encode information differently. We have begun to make a comparison between two of these modalities, force feedback spanning the elbow, and amplitude-modulated vibrotactile feedback, based on performance in a functional grasp and lift task. In randomly ordered trials, we assessed the performance of N=11 participants (8 able-bodied, 3 amputee) attempting to grasp and lift an object using an EMG controlled gripper under three feedback conditions (no feedback, vibrotactile feedback, and force feed-back), and two object weights that were undetectable by vision. Preliminary results indicate differences between able-bodied and amputee participants in coordination of grasp and lift forces. In addition, both force feedback and vibrotactile feedback contribute to significantly better task performance (fewer slips) and better adaptation following an unpredicted weight change. This suggests that the development and utilization of internal models for predictive control is more intuitive in the presence of haptic feedback.

Haptic cues take on meaning as a function of the context in which they are experienced. In interaction with objects in the physical environment, the context always includes a mechanical contact, at which point force and motion variables can be identified, and across which power may flow. In interaction with objects in a virtual or remote environment, it is not necessary for the contact across which haptic responses are rendered to be the same as the contact at which exploratory actions are applied. In this paper, we ask whether force/motion coupling has a significant impact on manual performance or cognitive load. We conducted an experiment in which n=7 participants attempted, while acting through a teleoperator, to discriminate three objects by their stiffness under two conditions. In one condition physical force/motion coupling was present, in the other it was not. To assess cognitive load, we engaged participants in a simultaneous cognitive task that included a response time measure. Results indicated no difference in manual discrimination performance. After rejecting the datasets of three of our participants based on inconsistent strategy by condition, we observed a small, non-significant trend toward lower cognitive load in the condition with physical coupling. Establishing a robust trend will require additional participants. While results are preliminary, we offer our paradigm as an important direction for new inquiry into the distinctions and interrelationships between information and its presentation in various haptic interface applications. Our work is aimed in particular at developing haptic feedback for use in prosthetic applications.