Online surgical phase recognition plays a significant role towards building contextual tools that could quantify performance and oversee the execution of surgical workflows. Current approaches are limited since they train spatial feature extractors using frame-level supervision that could lead to incorrect predictions due to similar frames appearing at different phases, and poorly fuse local and global features due to computational constraints which can affect the analysis of long videos commonly encountered in surgical interventions. In this paper, we present a two-stage method, called Long Video Transformer (LoViT), emphasizing the development of a temporally-rich spatial feature extractor and a phase transition map. The temporally-rich spatial feature extractor is designed to capture critical temporal information within the surgical video frames. The phase transition map provides essential insights into the dynamic transitions between different surgical phases. LoViT combines these innovations with a multiscale temporal aggregator consisting of two cascaded L-Trans modules based on self-attention, followed by a G-Informer module based on ProbSparse self-attention for processing global temporal information. The multi-scale temporal head then leverages the temporally-rich spatial features and phase transition map to classify surgical phases using phase transition-aware supervision. Our approach outperforms state-of-the-art methods on the Cholec80 and AutoLaparo datasets consistently. Compared to Trans-SVNet, LoViT achieves a 2.4 pp (percentage point) improvement in video-level accuracy on Cholec80 and a 3.1 pp improvement on AutoLaparo. Our results demonstrate the effectiveness of our approach in achieving state-of-the-art performance of surgical phase recognition on two datasets of different surgical procedures and temporal sequencing characteristics. The project page is available at https://github.com/MRUIL/LoViT.

This paper introduces SKiT, a fast Key information Transformer for phase recognition of videos. Unlike previous methods that rely on complex models to capture long-term temporal information, SKiT accurately recognizes high-level stages of videos using an efficient key pooling operation. This operation records important key information by retaining the maximum value recorded from the beginning up to the current video frame, with a time complexity of ${\mathcal{O}}\left( 1 \right)$. Experimental results on Cholec80 and AutoLaparo surgical datasets demonstrate the ability of our model to recognize phases in an online manner. SKiT achieves higher performance than state-of-the-art methods with an accuracy of 92.5% and 82.9% on Cholec80 and AutoLaparo, respectively, while running the temporal model eight times faster (7ms v.s. 55ms) than LoViT, which uses ProbSparse to capture global information. We highlight that the inference time of SKiT is constant, and independent from the input length, making it a stable choice for keeping a record of important global information, that appears on long surgical videos, essential for phase recognition. To sum up, we propose an effective and efficient model for surgical phase recognition that leverages key global information. This has an intrinsic value when performing this task in an online manner on long surgical videos for stable real-time surgical recognition systems.

This paper presents salient features of human-human interaction where one person with limited auditory and visual perception of the environment (a follower) is guided by an agent with full perceptual capabilities (a guider) via a hard rein along a given path. We investigate several salient features of the interaction between the guider and follower such as: 1) the order of an autoregressive (AR) control policy that maps states of the follower to actions of the guider; 2) how the guider may modulate the pulling force in response to the trust level of the follower; and 3) how learning may successively apportion the responsibility of control across different muscles of the guider. Based on experimental systems identification on human demonstrations from ten pairs of naive subjects, we show that guiders tend to adopt a third-order AR predictive control policy and followers tend to adopt second-order reactive control policy. Moreover, the extracted guider's control policy was implemented and validated by human-robot interaction experiments. By modeling the follower's dynamics with a time varying virtual damped inertial system, we found that it is the coefficient of virtual damping which is most sensitive to the trust level of the follower. We used these experimental insights to derive a novel controller that integrates an optimal order control policy with a push/pull force modulator in response to the trust level of the follower monitored using a time varying virtual damped inertial model.

The absence of touch of sense is a widely known drawback of robotic minimally invasive surgery (MIS). This paper proposes a design of optic soft tactile arrays which is promising to be adapted for MIS. The proposed design consists of multiple soft material channels. Each channel is designed using the Bernoulli pipe structure to amplify the sensor's sensitivity through input and output diameter difference. A multi-core optic fiber cable and a camera are used to capture the change of light intensity caused by the contact forces applied onto the individual soft material channels. The proposed sensor has the following advantages: 1) making use of 3D printing and soft material casting, it is suitable for designing sensors with high density of tactile elements; 2) it also allows the sensor to be designed in an arbitrary shape to fit various MIS applications; 3) compared to other light-intensity based tactile sensor, it is easy to fabricate and miniaturize; it avoids the complexity of attaching reflectors to individual sensing elements; 4) it is immune to electromagnetic interference. In this paper, a prototype which has 3×3 tactile elements in an area of 9.5 × 11 mm2has been developed and test for feasibility study. Also, a noise-filtering algorithm is developed to reduce the imaging noise. Validation experiments were carried out and results show that the average measurable force range for a single tactile element is 0 to 1.622N with an average accuracy of 97%. The sensor has low crosstalk-to-signal ratio, 1.8% on average, and has no signal drift over time.

Current surgical tele-manipulators do not provide explicit haptic feedback during soft tissue palpation. Haptic information could improve the clinical outcomes significantly and help to detect hard inclusions within soft-tissue organs indicating potential abnormalities. However, system instability is often caught by direct force feedback. In this paper, a new approach to intra-operative tumor localization is introduced. A virtual-environment tissue model is created based on the reconstructed surface of a soft-tissue organ using a Kinect depth sensor and the organ's stiffness distribution acquired during rolling indentation measurements. Palpation applied to this tissue model is haptically fed back to the user. In contrast to previous work, our method avoids the control issues inherent to systems that provide direct force feedback. We demonstrate the feasibility of this method by evaluating the performance of our tumor localization method on a soft tissue phantom containing buried stiff nodules. Results show that participants can identify the embedded tumors; the proposed method performed nearly as well as manual palpation.

Robotic manipulators for Robot-assisted Minimally Invasive Surgery (RMIS) pass through small incisions into the patient's body and interact with soft internal organs. The performance of traditional robotic manipulators such as the da Vinci Robotic System is limited due to insufficient flexibility of the manipulator and lack of haptic feedback. Modern surgical manipulators have taken inspiration from biology e.g. snakes or the octopus. In order for such soft and flexible arms to reconfigure itself and to control its pose with respect to organs as well as to provide haptic feedback to the surgeon, tactile sensors can be integrated with the robot's flexible structure. The work presented here takes inspiration from another area of biology: cucumber tendrils have shown to be ideal tactile sensors for the plant that they are associated with providing useful environmental information during the plant's growth. Incorporating the sensing principles of cucumber tendrils, we have created miniature sensing elements that can be distributed across the surface of soft manipulators to form a sensor network capable of acquire tactile information. Each sensing element is a retractable hemispherical tactile measuring applied pressure. The actual sensing principle chosen for each tactile makes use of optic fibres that transfer light signals modulated by the applied pressure from the sensing element to the proximal end of the robot arm. In this paper, we describe the design and structure of the sensor system, the results of an analysis using Finite Element Modeling in ABAQUS as well as sensor calibration and experimental results. Due to the simple structure of the proposed tactile sensor element, it is miniaturisable and suitable for MIS. An important contribution of this work is that the developed sensor system can be ”loosely” integrated with a soft arm effectively operating independently of the arm and without affecting the arm's motion during bending or elongation.

Advanced tactile tools in minimally invasive surgery have become a pressing need in order to reduce time and improve accuracy in localizing potential tissue abnormalities. In this regard, one of the main challenges is to be able to estimate tissue parameters in real time. In palpation, tactile information felt at a given location is identified by the viscoelastic dynamics of the neighboring tissue. Due to this reason the tissue examination behavior and the distribution of viscoelastic parameters in tissue should be considered in conjunction. This paper investigates the salient features of palpation behavior on soft tissue determining the effectiveness of localizing hard nodules. Experimental studies involving human participants, and validation tests using finite element simulations and a tele-manipulator, were carried out. Two distinctive tissue examination strategies in force-velocity modulation for the given properties of target tissue were found. Experimental results suggest that force-velocity modulations during continuous path measurements are playing an important role in the process of mechanical soft tissue examination. These behavioral insights, validated by detailed numerical models and robotic experimentations shed light on future designs of optimal robotic palpation.

In the case of human intervention in disaster response operations like indoor firefighting, where the environment perception is limited due to thick smoke, noise in the oxygen masks and clutter, not only limit the environmental perception of the human responders, but also causes distress. An intelligent agent (man/machine) with full environment perceptual capabilities is an alternative to enhance navigation in such unfavorable environments. Since haptic communication is the least affected mode of communication in such cases, we consider human demonstrations to use a hard rein to guide blindfolded followers with auditory distraction to be a good paradigm to extract salient features of guiding using hard reins. Based on numerical simulations and experimental systems identification based on demonstrations from eight pairs of human subjects, we show that, the relationship between the orientation difference between the follower and the guider, and the lateral swing patterns of the hard rein by the guider can be explained by a novel 3rdorder auto regressive predictive controller. Moreover, by modeling the two party voluntary movement dynamics using a virtual damped inertial model, we were able to model the mutual trust between two parties. In the future, the novel controller extracted based on human demonstrations can be tested on a human-robot interaction scenario to guide a visually impaired person in various applications like fire fighting, search and rescue, medical surgery, etc.

Robot-assisted minimal invasive surgery made it possible to improve the quality of surgical procedures and to enhance clinical outcomes. However, the need to palpate soft tissue organs with the aim to localize potential sites of abnormalities in real time has been recognized. For this work, ten subjects were recruited to perform a remote palpation procedure on a silicone phantom utilizing a tele-manipulation setup, to study their behavior when remotely palpating soft tissue. The stiffness values acquired during the remote palpation of a silicone phantom were transferred to the subjects by means of haptic and visual feedback. Participating subjects were asked to detect hard nodules in the silicone tissue using two distinct strategies: a) randomly chosen movements, and b) trajectory pattern, based on manual palpation techniques for clinical breast examination. We have compared relevant parameters, defining patterns observed during manual palpation, with the counterpart patterns occurring during remote palpation. The results show the effectiveness of applying palpation trajectory pattern used during manual soft tissue examination to tele-manipulation palpation.

During open surgery, surgeons can perceive the locations of tumors inside soft-tissue organs using their fingers. Palpating an organ, surgeons acquire distributed pressure (tactile) information that can be interpreted as stiffness distribution across the organ -an important aid in detecting buried tumors in otherwise healthy tissue. Previous research has focused on haptic systems to feedback the tactile sensation experienced during palpation to the surgeon during minimally invasive. However, the control complexity and high cost of tactile actuators limits its current application. This paper describes a pneumatic multi-fingered haptic feedback system for robot-assisted minimally invasive surgery. It simulates soft tissue stiffness by changing the pressure of an air balloon and recreates the deformation of fingers as experienced during palpation. The pneumatic haptic feedback actuator is validated by using finite element analysis. The results prove that the interaction stress between the fingertip and the soft tissue as well as the deformation of fingertips during palpation can be recreated by using our pneumatic multi-fingered haptic feedback method.

The aim of this paper is to improve the optimality and accuracy of techniques to guide a human in limited visibility and auditory conditions such as in fire-fighting in warehouses or similar environments. At present, breathing apparatus (BA) wearing fire-fighters move in teams following walls. Due to limited visibility and high noise in the oxygen masks, they predominantly depend on haptic communication through reins. An intelligent agent (man/machine) with full environment perceptual capabilities is an alternative to enhance navigation in such unfavorable environments, just like a dog guiding a blind person. This paper proposes an optimal state-dependent control policy to guide a follower with limited environmental perception, by an intelligent and environmentally perceptive agent. Based on experimental systems identification and numerical simulations on human demonstrations from eight pairs of participants, we show that the guiding agent and the follower experience learning for a optimal stable state-dependent novel 3rd and 2nd order auto regressive predictive and reactive control policies respectively. Our findings provide a novel theoretical basis to design advanced human-robot interaction algorithms in a variety of cases that require the assistance of a robot to perceive the environment by a human counterpart.

Accurate understanding of equipment tracking error is essential for decision making in image guided surgery. For tools tracked using markers attached to a rigid body, existing error estimation methods use the assumption that the individual marker errors are independent random variables. This assumption is not valid for all tracking systems. This paper presents a method to estimate a more accurate tracking error function, consisting of a systematic and random component. The proposed method does not require detailed knowledge of the tracking system physics. Results from a pointer calibration are used to demonstrate that the proposed method provides a better match to observed results than the existing state of the art. A simulation of the pointer calibration process is then used to show that existing methods can underestimate the pointer calibration error by a factor of two. A further simulation of laparoscopic camera tracking is used to show that existing methods cannot model important variations in system performance due to the angular arrangement of the tracking markers. By arranging the markers such that the systematic errors are nearly identical for all markers, the rotational component of the tracking error can be reduced, resulting in a significant reduction in target tracking errors.

Maintaining the grip on an artery with a pulsating impedance, holding the steering wheel of a vehicle on a bumpy terrain, or holding a live hamster without excessive squeezing may be trivial tasks to most humans. However, a robot will find it very difficult to maintain the grip of such uncertain objects based on real-time feedback control. This paper presents a stochastic control law to maintain the grip on an uncertain object while manipulating against external forces. The radial impedance parameters of the soft object is assumed to undergo Gaussian random variations. Here we demonstrate that the proposed model free grip controller can maintain a safe grip at two diagonally opposite points of the object merely based on the statistics of the normal force. It accomplishes this by computing a probability of grip failure to adapt the compression on the soft object. A novel optimal estimation algorithm that can concurrently estimate the unknown impedance parameters of the object and the states of the coupled dynamic system is discussed as a potential tool to be used in predictive optimal impedance control on uncertain objects. Experimental results on adaptive grip control on a cylindrical tube inflated and deflated with a Gaussian random variation has been presented to validate the algorithm.

Robotic manipulators for minimally invasive surgeries have traditionally been rigid, with a steerable end effector. While the rigidity of manipulators improve precision and controllability, it limits reachability and dexterity in constrained environments. Soft manipulators with controllable stiffness on the other hand, can be deployed in single port or natural orifice surgical applications to reach a wide range of areas inside the body, while being able to passively adapt to uncertain external forces, adapt the stiffness distribution to suit the kinematic and dynamic requirements of the task, and provide flexibility for configuration control. Here, we present the design of a snake-like laboratory made soft robot manipulator of 20 mm in average diameter, which can actuate, soften, or stiffen joints independently along the length of the manipulator by combining granular jamming with McKibben actuators. It presents a comprehensive study on the relative contributions of the granule size, material type, and membrane coupling on the range, profile, and variability of stiffness.

This paper describes the design and construction of a miniaturized triaxial force sensor which can be applied inside a magnetic resonance imaging (MRI) machine. The sensing principle of the sensor is based on an optical intensity modulation mechanism that utilizes bent-tip optical fibers to measure the deflection of a compliant platform when exposed to a force. By measuring the deflection of the platform using this optical approach, the magnitude and direction of three orthogonal force components (Fx, Fy, and Fz) can be determined. The sensor prototype described in this paper demonstrates that it can perform force measurements in axial and radial directions with working ranges of +/-2 N. Since the sensor is small in size and entirely made of nonmetallic materials, it is compatible with minimally invasive surgery (MIS) and safe to be deployed within magnetic resonance (MR) environments.