In cable-driven robots, the actuation is transmitted via long cables for control and manipulation of the end-effectors. The cables introduce biases due to non-linear tension, creating a challenge for accurate modeling and localization. This paper presents a novel data-driven algorithm that addresses these biases in partially known dynamic systems, focusing on cable-driven robots. The proposed approach integrates classic Kalman filtering with deep learning to enhance tracking and overcome the limitations of existing state-space modeling. Our algorithm learns from data to explicitly track the robot’s end-effector, while implicitly tracking its cable-induced bias. We demonstrate the effectiveness of the algorithm on two different robotic manipulators: a planar two-link robotic manipulator, and the Raven surgical robot, constituting testbeds for dynamic systems where the bias arises from time-varying cable tension. Our experimental results reveal that while conventional methods like the Extended Kalman Filter struggle with pose tracking due to system biases, our algorithm successfully localizes the robot’s end-effector and uncovers underlying biases, showing significant advancements in predictive capabilities and dynamic system understanding.

In robot-assisted minimally invasive surgery, optimal camera positioning is crucial for effective visualization and manipulation of tissues, which impacts the success of procedures. Traditional camera control can increase cognitive workload, lead to suboptimal camera viewpoints, and complicate surgical tasks. We propose an autonomous camera system that uses situational awareness and real-time prediction of user intent from kinematic data, and adjusts the camera position dynamically during a simulated suturing task. Such a system reduces the need to manually adjust the camera, allowing users to stay focused on the procedure. We demonstrated the framework in a user study with eight non-expert participants. They used the da Vinci Research Kit to control a simulated camera and instruments in a suturing task. We compared the performances in the suturing task with the autonomous and the teleoperated camera control. The autonomous system reduced execution time by 43% , shortened path length by 30% , and decreased completion cost by 35%. These results serve as a proof of concept for a situationally aware camera system and suggest that autonomous camera control can improve efficiency and simplify surgical workflow.

Computational approaches to biological motor control are used to discover the building blocks of human motor behaviour. Models explaining features of human hand movements have been studied thoroughly, yet only a few studies attempted to explain the control of the orientation of the hand; instead, they mainly focus on the control of hand translation, predominantly in a single plane. In this study, we present a new methodology to study the way humans control the orientation of their hands in three dimensions and demonstrate it in two sequential experiments. We developed a quaternion-based score that quantifies the geodicity of rotational hand movements and evaluated it experimentally. In the first experiment, participants performed a simple orientation-matching task with a robotic manipulator. We found that rotations are generally performed by following a geodesic in the quaternion hypersphere, which suggests that, similarly to translation, the orientation of the hand is centrally controlled, possibly by optimizing geometrical properties of the hand's rotation. This result established a baseline for the study of human response to perturbed visual feedback of the orientation of the hand. In the second experiment, we developed a novel visuomotor rotation task in which the rotation is applied on the hand's rotation, and studied the adaptation of participants to this rotation, and the transfer of the adaptation to a different initial orientation. We observed partial adaptation to the rotation. The patterns of the transfer of the adaptation to a different initial orientation were consistent with the representation of the orientation in extrinsic coordinates. The methodology that we developed allows for studying the control of a rigid body without reducing the dimensionality of the task. The results of the two experiments open questions for future studies regarding the mechanisms underlying the central control of hand orientation. These results can be of benefit for many applications that involve fine manipulation of rigid bodies, such as teleoperation and neurorehabilitation.

OBJECTIVE: Robotic-assisted minimally invasive surgery (RAMIS) became a common practice in modern medicine and is widely studied. Surgical procedures require prolonged and complex movements; therefore, classifying surgical gestures could be helpful to characterize surgeon performance. The public release of the JIGSAWS dataset facilitates the development of classification algorithms; however, it is not known how algorithms trained on dry-lab data generalize to real surgical situations. METHODS: We trained a Long Short-Term Memory (LSTM) network for the classification of dry lab and clinical-like data into gestures. RESULTS: We show that a network that was trained on the JIGSAWS data does not generalize well to other dry-lab data and to clinical-like data. Using rotation augmentation improves performance on dry-lab tasks, but fails to improve the performance on clinical-like data. However, using the same network architecture, adding the six joint angles of the patient-side manipulators (PSMs) features, and training the network on the clinical-like data together lead to notable improvement in the classification of the clinical-like data. DISCUSSION: Using the JIGSAWS dataset alone is insufficient for training a gesture classification network for clinical data. However, it can be very informative for determining the architecture of the network, and with training on a small sample of clinical data, can lead to acceptable classification performance. SIGNIFICANCE: Developing efficient algorithms for gesture classification in clinical surgical data is expected to advance understanding of surgeon sensorimotor control in RAMIS, the automation of surgical skill evaluation, and the automation of surgery.

Robotic-Assisted Minimally Invasive Surgery allows for easy recording of kinematic data, and presents excellent opportunities for data-intensive approaches to assessment of surgical skill, system design, and automation of procedures. However, typical surgical cases result in long data streams, and therefore, automated segmentation into gestures is important. The public release of the JIGSAWS dataset allowed for developing and benchmarking data-intensive segmentation algorithms. However, this dataset is small and the gestures are similar in their structure and directions. This may limit the generalization of the algorithms to real surgical data that are characterized by movements in arbitrary directions. In this paper, we use a recurrent neural network to segment a suturing task, and demonstrate one such generalization problem-limited generalization to rotation. We propose a simple augmentation that can solve this problem without collecting new data, and demonstrate its benefit using: (1) the JIGSAWS dataset, and (2) a new dataset that we recorded with a da Vinci Research Kit. Our study highlights the prospect of using data augmentation in the analysis of kinematic data in surgical data science.

In this paper, we investigate the grasping of rigid objects in a unilateral robot-assisted minimally invasive surgery (RAMIS). We define a human-centered transparency that quantifies the natural action and perception in RAMIS. We demonstrate this human-centered transparency analysis for different values of gripper scaling-the scaling between the grasp aperture of the surgeon-side manipulator and the aperture of the surgical instrument grasper. A total of 31 participants performed teleoperated grasping and perceptual assessment of rigid objects in one of three gripper scaling conditions (fine, normal, and quick, trading off precision and responsiveness). A psychophysical analysis of the variability of the maximal grasping aperture during prehension and of the reported size of the object revealed that under normal and quick (but not under the fine) gripper scaling conditions, the teleoperated grasping with our system was similar to natural grasping and, therefore, human-centered transparent. We anticipate that using motor control and psychophysics for human-centered optimization of teleoperation control will eventually improve the usability of RAMIS.

Force feedback in tool-mediated interactions with the environment is important for successful performance of complex tasks in our daily life as well as in specialized fields like medicine. Stylus-based haptic devices are studied and used extensively, and most of these devices require either grounding or attachment to the body of the user. Recently, non-grounded haptic devices are getting an increasing attention. In this paper, we propose a novel method to represent the vertical forces that are applied on the tip of a tool: a non-grounded rotation mechanism that mimics the cutaneous sensation that is caused by these tool-tip forces. To evaluate this method, we developed a novel ungrounded haptic device - HapSticks - that renders the sensation of manipulating objects using chopsticks. First, we present the novel mechanism, and test the pressure that it applies on the hand of the user when rendering a force at the tip of the tool in comparison to applying a real force at the tip of the tool. Next, we used the mechanism to build the HapSticks device as an example of an application of the proposed method, and present a psychophysical evaluation of this device in a virtual weight discrimination task.

During robotic teleoperation, the dynamics of the master manipulators and the control of remote-side instruments impose challenges on the motor system of the human operator, and may impact performance and learning. In teleoperated robot-assisted minimally invasive surgery, there is a clear correlation between patient outcomes and the surgeon's case experience. However, the effect of the teleoperator on human motor skills and the relationship between these motor skills and patient outcomes are unknown. We used the da Vinci Research Kit, a custom research version of the da Vinci Surgical System, to compare teleoperated and open needle-driving movements of experienced da Vinci surgeons and novices. The experimental protocol consisted of structured but unconstrained needle driving trials repeated 80 times to allow for computational modeling of movement coordination and learning. Kinematic analysis showed that teleoperation increases trial time but reduces path length, that the trial times and path lengths of experienced surgeons are smaller than those of novices. In addition, there are significant differences in learning between experienced surgeons and novice users. Modeling of the movements and learning processes of experienced and novice surgeons may be used in the design of novel controllers that will expand robotic surgery capabilities and improve robot-assisted surgical skill acquisition.

When a person interacts with an environment using a tool, he/she receives tactile information in the form of fingerpad skin deformation. Different interaction forces and torques on the tool cause different skin deformation patterns on the fingerpads. We designed a 6-degree-of-freedom tactile device that creates similar skin deformation patterns on the fingerpads. The device communicates force and torque information by translating and rotating skin deformation tactors relative to the fingerpads. An experiment was conducted to determine participants' ability to use skin deformation tactile cues to perform a peg-in-hole insertion task. Results show that participants can use the tactile cues to reduce interaction force and torque, and they use the tactile force cues to reduce interaction force more than they use the tactile torque cues to reduce interaction torque. Rendering force and torque cues simultaneously causes device saturation and degrades user performance. These results suggest that additional training may help participants use the skin deformation torque cues, and motivate a tactile device design that decouples force and torque skin deformation rendering to minimize device saturation. Fingerpad skin deformation is a promising form of tactile feedback to convey force and torque information in teleoperation systems such as robot-assisted surgery, where force feedback may be undesirable due to stability and safety concerns.

When we use a tool to explore or manipulate an object, friction between the surface of the tool and the fingerpads generates skin stretch cues that are related to the interaction forces between the tool and the object. In this study, we emulate these naturally occurring skin stretch cues in order to convey force direction and magnitude information to users during teleoperation. We hypothesize that skin stretch feedback is a useful substitute for kinesthetic force feedback in force-sensitive teleoperated tasks. In this study, ten participants performed teleoperated palpation to determine the orientation of a stiff region in a surrounding artificial tissue using five feedback conditions: skin stretch, force, reduced gain force, graphic, and vibration. When participants received skin stretch feedback, they localized the stiff region as well as with force feedback, with no increase in task completion time. Additionally, participants receiving skin-stretch feedback localized the stiff region statistically significantly more accurately than those using vibration feedback. Although participants using skin stretch exhibited higher interaction forces than when using force, vibration, and graphical feedback, skin stretch statistically significantly decreased interaction forces compared with reduced gain force feedback. Thus, skin-stretch feedback is a compelling substitute for force feedback and may be useful in scenarios where force feedback is reduced or infeasible.

The Active Handrest is a large workspace assistive robotic device that has been shown to significantly improve the user performance of various movement tasks. This device has the potential to assist highly accurate motions without the need for a robotically enabled or comanipulated tool while allowing the user to retain complete control over a grasped tool. This paper evaluates the Active Handrest's effect on the kinematics of user movements together with various virtual fixture control strategies to further improve the human performance while using the Active Handrest. The most effective fixture strategy for tracing straight lines is first determined. Then, this fixture strategy is directly implemented on the Active Handrest's motions. Finally, the performance of different fixture strategies is assessed through speed-accuracy tradeoff and movement kinematics. It is concluded that fixtures implemented on the Active Handrest are as effective as those implemented on a grasped tool, which enables users of the Active Handrest high accuracy with ordinary tools such as a scalpel or paint brush. The performance is further improved when fixtures are implemented on both the Active Handrest and the grasped tool.

During tool-mediated interactions with objects, we experience force and fingerpad skin stretch resulting from shear forces caused by friction between the fingerpad skin and the stylus. When probing an object, for the same penetration distance, a stiffer object causes a larger load force and, thus, greater fingerpad skin stretch. We hypothesized that rendering additional artificial skin stretch together with force will increase perceived stiffness. We created a Skin Stretch Stylus that renders skin stretch through tactor displacement, attached it to a force-feedback device, and performed a study to characterize the effect of tactor displacement-induced skin stretch on stiffness perception. Results showed that adding artificial skin stretch causes additive augmentation of perceived stiffness across a range of surface stiffness, and the addition is a linear function of tactor displacement gain. However, intersubject variability in the estimated slope coefficient was large. We propose a model that explains the additive effect and suggests potential sources for the intersubject variability. We conclude that augmenting force feedback with skin stretch can increase users' perception of stiffness, but the effect is user-specific. Such augmentation may be useful in virtual environment and teleoperation scenarios when force feedback gains must be kept low to prevent feedback-induced instabilities, or when force feedback is limited due to actuator force limits.

When interacting with everyday objects, we experience kinesthetic force feedback as well as various forms of cutaneous tactile feed-back. Skin stretch is part of the cutaneous tactile experience that is caused by friction between the skin and the grasped object. Interacting with stiffer objects causes larger force, and results in a larger amount of skin stretch. Therefore, we hypothesize that adding artificial fingerpad skin stretch to kinesthetic force feedback will increase users' perception of stiffness. A tactile display called the Skin Stretch Stylus was designed to augment kinesthetic force feedback with skin stretch feedback. The change in users' stiffness perception due to the addition of skin stretch feedback is quantified through a two-alternative forced-choice paradigm, method of constant stimuli experiment. In this experiment, subjects compared the stiffness of virtual springs with kinesthetic force feedback augmented with skin stretch feedback versus virtual springs with only kinesthetic force feedback. Results show that the addition of skin stretch causes a significant increase in the perception of stiffness, and this effect increases with higher amount of applied skin stretch. These results indicate that skin stretch feedback could be used to augment perceived stiffness in situations where it is not possible to increase force feedback gains. Such scenarios include teleoperation systems where force feedback gains must remain low to ensure stability, and haptic devices with limited actuator force.

Teleoperated robot-assisted surgery (RAS) provides surgeons with improved dexterity, movement control, and visualization in comparison to standard minimally invasive surgery. However, there exists little quantitative understanding of the motor performance of human operators in RAS. Models of how users control their movements and how this control relates to surgical performance could provide inspiration for new robot or human interface designs, as well as more targeted training methods. Toward this end, we present a framework for the analysis of surgeon arm posture variability based on the uncontrolled manifold (UCM) concept, a method used in the study of human motor control for testing hypotheses about the coupling of control and task variables. We partition users' joint angle variability into variability that does and does not result in hand trajectory change. In a preliminary study applying this framework, we explored how expert and novice operators control planar reaching and reversal movements when moving freehand as well as using a teleoperated RAS system. We show that only movements in task-relevant directions are stabilized by the coordination of joint angles, and that this stabilization is stronger for expert movements than novice movements. We also show that stabilization is stronger in freehand than teleoperated movements, especially for the expert. These preliminary findings suggest that the proposed framework can be useful for: (1) assessment of teleoperator design and control that reveals how design parameters affect the ability of the user to exploit the UCM for stabilization of hand movement, and (2) skill assessment in RAS.

Skin stretch is a novel haptic feedback method that can provide a human operator with information about the magnitude and direction of an applied force. To evaluate the potential for skin stretch feedback to be used as a sensory substitute for kinesthetic (force) feedback in robotic teleoperation systems, a study was conducted to measure the ability of users to discriminate environment stiffness using varying levels of fingerpad skin stretch instead of force feedback. A new, high-fidelity skin stretch feedback device was developed that imposes tangential fingerpad skin stretch in proportion to the intended level of force feedback. In psychophysical experiments, users received skin stretch feedback with magnitude proportional to the users' penetration depth into a virtual wall. Users' stiffness discrimination capability using skin stretch was comparable to that of using force feedback. Furthermore, larger skin stretch cues were perceived by users as portraying greater stiffness without any advance training, which indicates that skin stretch feedback would be an intuitive sensory substitute for force feedback. Thus, skin stretch feedback is a promising method for conveying kinesthetic force information in applications such as robot-assisted surgery, where high levels of force feedback may not be desirable due to stability or safety concerns.

In bilateral teleoperation, a human operator manipulates a remote environment through a pair of master and slave robots. The transparency quantifies the fidelity of the teleoperation system, and is typically defined as the ability to accurately display remote environment properties to the operator. We propose a novel multidimensional measure of transparency which takes into account the human operator and consists of three components: 1) perceptual transparency, which quantifies human perception of the remote environment, 2) local motor transparency, which quantifies how far is the movement of the human operator from ideal, and 3) remote motor transparency, which describes how far is the movement of the remote device from ideal. We suggest that for many practical applications, the goal of the transparency optimization is to maintain perceptual and remote motor transparency while sacrificing local motor transparency, and that it is plausible to take advantage of the gap between perception and action in the operators sensorimotor system. We prove analytically that for a teleoperation channel with a position and force scaling and a constant transmission delay, in a palpation and perception of stiffness task, it is possible to find gains that ensure perfect perceptual and remote motor transparency while maintaining stability. We also show that stability depends on the operator that maintain sufficient arm impedance relative to environment impedance and delay.