A key component to the success of master-slave surgical systems is the quality of the master interface used to relay the surgeon's instructions to the slave robot. In previous work the authors developed a gaze-assisted intention recognition scheme, allowing the system to dynamically adapt the motion scaling based on where the user is trying to reach. This allowed users to perform tasks significantly more quickly and with less need for clutching. However, the system possessed a number of core parameters that were manually optimized, potentially providing a non-optimal solution depending on the user. This paper presents a Bayesian approach to the problem of optimizing a human-robot interface in a user-specific manner. Two Bayesian optimization methods are studied: one in which users are asked to grade robot behavior for a given set of parameters, and one where only preference relative to other parameter sets is expressed. The performance of these optimizations is evaluated in a blind comparison user study, demonstrating that the optimized parameters are preferred to the manually optimized ones in over 90 % of cases after only 12 test samples. These parameters are further shown to perform at least as well as the manually optimized ones in all cases, and showing statistically significant improvement in the case of the graded optimization.

Traditional robotic surgical systems rely entirely on robotic arms to triangulate articulated instruments inside the human anatomy. This configuration can be ill-suited for working in tight spaces or during single access approaches, where little to no triangulation between the instrument shafts is possible. The control of these instruments is further obstructed by ergonomic issues: The presence of motion scaling imposes the use of clutching mechanics to avoid the workspace limitations of master devices, and forces the user to choose between slow, precise movements, or fast, less accurate ones. This paper presents a bi-manual system using novel self-triangulating 6-degrees-of-freedom (DoF) tools through a flexible elbow, which are mounted on robotic arms. The control scheme for the resulting 9-DoF system is detailed, with particular emphasis placed on retaining maximum dexterity close to joint limits. Furthermore, this paper introduces the concept of gaze-assisted adaptive motion scaling. By combining eye tracking with hand motion and instrument information, the system is capable of inferring the user's destination and modifying the motion scaling accordingly. This safe, novel approach allows the user to quickly reach distant locations while retaining full precision for delicate manoeuvres. The performance and usability of this adaptive motion scaling is evaluated in a user study, showing a clear improvement in task completion speed and in the reduction of the need for clutching.

Advances in optical imaging, and probe-based Confocal Laser Endomicroscopy (pCLE) in particular, offer real-time cellular level information for in-vivo tissue characterization. However for large area coverage, the limited field-of-view necessitates the use of a technique known as mosaicking to generate usable information from the incoming image stream. Mosaicking also needs a continuous stream of good quality images, but this is challenging as the probe needs to be maintained within an optimal working range and the contact force controlled to minimize tissue deformation. Robotic manipulation presents a potential solution to these challenges, but the lack of haptic feedback in current surgical robot systems hinders the technology's clinical adoption. This paper proposes a sensorless alternative based on processing the incoming image stream and deriving a quantitative measure representative of the image quality. This measure is then used by a controller, designed using model-free reinforcement learning techniques, to maintain optimal contact autonomously. The developed controller has shown near real-time performance in overcoming typical loss-of-contact and excess-deformation scenarios experienced during endomicroscopy scanning procedures.

Effective in situ, in vivo tumour margin assessment is an important, yet unmet, clinical demand in surgical oncology. Recent advances in probe-based optical imaging tools such as confocal endomicroscopy is making inroads in clinical applications. In practice, maintaining consistent tissue contact whilst ensuring large area surveillance is crucial for its practical adoption and for this reason there is a great demand for robotic assistance so that high-speed endomicroscopes can be combined with autonomous scanning, thus simplifying its incorporation in routine surgical workflows. In this paper, a cooperatively controlled robotic manipulator is developed, which provides a stable mechatronically-enhanced platform for micro-scanning tools to perform local high resolution mosaics over 3D undulating moving surfaces. Detailed kinematic and overall system performance analyses are provided and the results demonstrate the adaptability in terms of both contact force and orientation control of the system, and thus its simplicity in practical deployment and value for clinical adoption.

Robot-assisted minimally invasive surgery can benefit from the automation of common, repetitive or well-defined but ergonomically difficult tasks. One such task is the scanning of a pick-up endomicroscopy probe over a complex, undulating tissue surface to enhance the effective field-of-view through video mosaicing. In this paper, the da Vinci®surgical robot, through the dVRK framework, is used for autonomous scanning and 2D mosaicing over a user-defined region of interest. To achieve the level of precision required for high quality mosaic generation, which relies on sufficient overlap between consecutive image frames, visual servoing is performed using a combination of a tracking marker attached to the probe and the endomicroscopy images themselves. The resulting sub-millimetre accuracy of the probe motion allows for the generation of large mosaics with minimal intervention from the surgeon. Images are streamed from the endomicroscope and overlaid live onto the surgeons view, while 2D mosaics are generated in real-time, and fused into a 3D stereo reconstruction of the surgical scene, thus providing intuitive visualisation and fusion of the multi-scale images. The system therefore offers significant potential to enhance surgical procedures, by providing the operator with cellular-scale information over a larger area than could typically be achieved by manual scanning.

The performance of a master-slave robotic system depends significantly on the ergonomics and the capability of its master device to correctly interface the user with the slave robot. Master manipulators generating commands in task space represent a commonly adopted solution for controlling a range of slave robots while retaining an ergonomic design. However, these devices present several drawbacks, such as requiring the use of clutching mechanics to compensate for the mismatch between slave and master workspaces, and the lack of capability to intuitively transmit important information such as specific joint limits to the user. In this paper, a novel joint-space master manipulator is presented. This manipulator emulates the kinematic structure of highly flexible surgical instruments which it is designed to control. This system uses 6 active degrees of freedom to compensate for its own weight, as well as to provide force feedback corresponding to the slave robot's joint limits. A force/torque sensor integrated at the end effector is used to relay user-generated forces and torques directly to specific joints. This is performed to counteract the friction stemming from structural constraints imposed by the kinematic design of the instruments. Finally, a usability study is carried out to test the validity of the system, proving that the instruments can be intuitively controlled even at the extremities of the workspace.

Tracking and shape estimation of flexible robots that navigate through the human anatomy are prerequisites to safe intracorporeal control. Despite extensive research in kinematic and dynamic modeling, inaccuracies and shape deformation of the robot due to unknown loads and collisions with the anatomy make shape sensing important for intraoperative navigation. To address this issue, vision-based solutions have been explored. The task of 2-D tracking and 3-D shape reconstruction of flexible robots as they reach deep-seated anatomical locations is challenging, since the image acquisition techniques usually suffer from low signal-to-noise ratio or slow temporal responses. Moreover, tracking and shape estimation are thus far treated independently despite their coupled relationship. This paper aims to address tracking and shape estimation in a unified framework based on Markov random fields. By using concentric tube robots as an example, the proposed algorithm fuses information extracted from standard monoplane X-ray fluoroscopy with the kinematics model to achieve joint 2-D tracking and 3-D shape estimation in realistic clinical scenarios. Detailed performance analyses of the results demonstrate the accuracy of the method for both tracking and shape reconstruction.

Transanal Endoscopic Microsurgery (TEM) is a minimally invasive oncological resection procedure that utilises a natural orifice approach rather than the traditional abdominal or open approach. However, TEM has a significant recurrence rate due to incomplete excisions, which can possibly be attributed to the absence of intraoperative image guidance. The use of real-time histological data could allow the surgeons to assess the surgical margins intraoperatively and adjust the procedure accordingly. This paper presents the integration of endomicroscopy and ultrasound imaging through a robotically actuated instrument. Endomicroscopy can provide high resolution images at a surface level while ultrasound provides depth resolved information at a macroscopic level. Endomicroscopy scanning is achieved with a novel scanning approach featuring a passive force adaptive mechanism. The instrument is manipulated across the surgical workspace through an articulated flexible shaft. This results in the ability to perform large area mosaics coupled with ultrasound scanning. In addition, the use of endoscopic tracking is demonstrated, allowing three-dimensional reconstruction of the ultrasound data displayed onto the endoscopic view. An ex vivo study on porcine colon tissue has been performed, demonstrating the clinical applicability of the instrument.

Recent work on probe-based confocal endomicroscopy has demonstrated its potential role for real-time assessment of tumour margins during breast conserving surgery. However, endomicroscope probes tend to have a very small field-of-view, making surveillance of large areas of tissue difficult, and limiting practical clinical deployment. In this paper, a new robotic device for controlled, large area scanning based on a fibre bundle endomicroscope probe is proposed. The prototype uses a 2-DOF mechanism (−90 to +90 degrees bending on one axis, 360 degrees of rotation on a second axis) as well as a passive linear structure to conform to undulating surfaces. Both axes are driven by brushless DC servo motors with computer control, thus facilitating large field-of-view mosaicing. Experimental results have shown good repeatability and low hysteresis of the device, which is able to scan different surface trajectories (e.g. a spiral pattern over a hemi-spherical surface) with consistent tissue contact. Ex vivo human breast tissue results are demonstrated, illustrating a viable scanning approach for breast endomicroscopy.

Recent technological advances in surgery have permitted cellular and molecular imaging to be carried out intra-operatively. Although optical biopsy techniques such as probe-based confocal laser endomicroscopy (pCLE) have enabled real-time diagnosis and tissue characterisation in vivo, the flexibility of the probe introduces significant challenges under manual control. Examination of large tissue areas is particularly challenging due to micron-scale resolution of the probe and the need for maintaining consistent probe orientation and force contact with the tissue to avoid cellular deformation or damage. The use of a robotic manipulator to perform surface scanning automatically introduces great benefits in terms of positioning repeatability and accuracy. However, pre-programming of such complex task is not realistic due to patient-specific anatomy and constant changes in tissue morphology during the operation. To overcome this problem, a cooperative, in situ microscopic scanning and simultaneous tissue surface reconstruction technique is proposed. The system provides a hands-on, learning-based framework for optimal trajectory coverage from surgeon-demonstrated motions intraoperatively. The position and force information acquired during the scanning are also used to simultaneously reconstruct the surface morphology and combined with the pCLE images to generate a 3D functional map of the tissue.

This paper presents an Implicit Active Constraints control framework for robot-assisted minimally invasive surgery. It extends on current frameworks by prescribing the external constraints implicitly from the operator motion, forgoing the need for pre-operative imaging; the constraints are defined in situ so as to avoid the use of invasive fiducial markers. A hands-on cooperatively-controlled robotic platform, comprising of a surgical instrument and a compliant manipulator, has been designed for an arthroscopic procedure. The surgical platform is capable of constraining the pose of the instrument so as to ensure it passes through the incision point and does not cause trauma to the surrounding tissue. A flexible arthroscopic instrument is designed and its use is investigated to enlarge reachable and dexterous workspace, increasing the accessibility to the target anatomy. The behaviour of the flexible instrument is analysed. A detailed performance analysis is conducted on a group of subjects for validating the control framework, simulating a minimally invasive arthroscopic procedure. Results demonstrate a statistically significant enhancement in the control ergonomics as well as the accuracy and safety of the procedure.

We propose a learning-based controller to enable autonomous execution of the eFAST scanning by a lightweight robotic manipulator according to expert demonstrations. The benefits of this approach are two-fold. Firstly, the automatically acquired USS images can be sent to the expert radiologist from a remote location without the need for complex robotic tele-operation. Secondly, the application of learning by demonstration alleviates the complexity of robotic programming and allows extracting operator-specific knowledge in situ in a natural and intuitive way. The provision of incorporating force information can further improve the versatility of the system, allowing easy adaptation to different dynamic environments.