Precise pedicle screw placement is crucial in spine surgery, where minor inaccuracies can result in significant complications. Despite introducing robot-assisted navigation systems to aid surgeons, accommodating the spine’s nonrigid movements (due to patient movement or interactions with the surgeon) often necessitates repeated intraoperative imaging, leading to increased radiation exposure. To address this challenge, we propose a novel method that utilizes the functional map (FM) framework to transfer drilling trajectories from preoperative CT scans to partially observed and noisy spine models. Specifically, the FM correspondences enhance the registration quality of pre-operative and perioperative 3D spine model data, even in the presence of non-rigid deformations. Through comprehensive simulations, we assess the method’s effectiveness across various cases of complex deformations using a spine model consisting of five lumbar vertebrae obtained through CT scans. Validation involves evaluating registration errors in translation and rotation and verifying the clinical validity of transferred drilling trajectories. The results demonstrate the method’s efficiency in transferring drilling trajectories onto noisy, partially observed, and deformed spine models.

This paper presents a spectral domain registration-based visual servoing scheme that works on 3D point clouds. Specifically, we propose a 3D model/point cloud alignment method, which works by finding a global transformation between reference and target point clouds using spectral analysis. A 3D Fast Fourier Transform (FFT) in$\mathbb{R}^{3}$is used for the translation estimation, and the real spherical harmonics in$\boldsymbol{SO}(3)$are used for the rotations estimation. Such an approach allows us to derive a decoupled 6 degrees of freedom (DoF) controller, where we use gradient ascent optimisation to minimise translation and rotational costs. We then show how this methodology can be used to regulate a robot arm to perform a positioning task. In contrast to the existing state-of-the-art depth-based visual servoing methods that either require dense depth maps or dense point clouds, our method works well with partial point clouds and can effectively handle larger transformations between the reference and the target positions. Furthermore, the use of spectral data (instead of spatial data) for transformation estimation makes our method robust to sensor-induced noise and partial occlusions. We validate our approach by performing experiments using point clouds acquired by a robot-mounted depth camera. Obtained results demonstrate the effectiveness of our visual servoing approach.

Efficiently exploiting multi-modal inputs for accurate RGB-D saliency detection is a topic of high interest. Most existing works leverage cross-modal interactions to fuse the two streams of RGB-D for intermediate features' enhancement. In this process, a practical aspect of the low quality of the available depths has not been fully considered yet. In this work, we aim for RGB-D saliency detection that is robust to the low-quality depths which primarily appear in two forms: inaccuracy due to noise and the misalignment to RGB. To this end, we propose a robust RGB-D fusion method that benefits from (1) layer-wise, and (2) trident spatial, attention mechanisms. On the one hand, layer-wise attention (LWA) learns the trade-off between early and late fusion of RGB and depth features, depending upon the depth accuracy. On the other hand, trident spatial attention (TSA) aggregates the features from a wider spatial context to address the depth misalignment problem. The proposed LWA and TSA mechanisms allow us to efficiently exploit the multi-modal inputs for saliency detection while being robust against low-quality depths. Our experiments on five bench-mark datasets demonstrate that the proposed fusion method performs consistently better than the state-of-the-art fusion alternatives. The source code is publicly available at: https://github.com/Zongwei97/RFnet.

Handling object deformations for robotic grasping is still a major problem to solve. In this paper, we propose an efficient learning-free solution for this problem where generated grasp hypotheses of a region of an object are adapted to its deformed configurations. To this end, we investigate the applicability of functional map (FM) correspondence, where the shape matching problem is treated as searching for correspondences between geometric functions in a reduced basis. For a user selected region of an object, a ranked list of grasp candidates is generated with local contact moment (LoCoMo) based grasp planner. The proposed FM-based methodology maps these candidates to an instance of the object that has suffered arbitrary level of deformation. The best grasp, by analysing its kinematic feasibility while respecting the original finger configuration as much as possible, is then executed on the object. We have compared the performance of our method with two different state-of-the-art correspondence mapping techniques in terms of grasp stability and region grasping accuracy for 4 different objects with 5 different deformations.

This paper deals with the control of a redundant robotic system for middle ear surgery (i.e., cholesteatoma tissues removal). The targeted robotic system is a macro-micro-scale robot composed of a redundant seven degrees of freedom (DoFs) on which is attached a two DoFs robotized flexible fiberscope. Two different control architectures are proposed to achieve a defined surgical procedure to remove the pathological tissue inside the middle ear cavity. The first proposed control mode is based on the position-based tele-operation of the entire system using a joystick (Phantom Omni) as a master arm. The second one combines comanipulation of the seven DoFs robotic arm using an embedded force/torque sensor and an end-frame tele-operation of the remaining two DoFs fiberscope using a lab-made in-hand joystick. Experimental validation is performed to evaluate and compare the performance of both developed control schemes. The obtained results using the lab-made platform and the proposed controllers are discussed.

This paper deals with the control of a laser spot in the context of minimally invasive surgery of the middle ear, e.g., cholesteatoma removal. More precisely, our work is concerned with the exhaustive burring of residual infected cells after primary mechanical resection of the pathological tissues since the latter cannot guarantee the treatment of all the infected tissues, the remaining infected cells cause regeneration of the diseases in 20%-25 % of cases, which require a second surgery 12–18 months later. To tackle such a complex surgery, we have developed a robotic platform that consists of the combination of a macro-scale system (7 degrees of freedoms (DoFs) robotic arm) and a micro-scale flexible system (2 DoFs) which operates inside the middle ear cavity. To be able to treat the residual cholesteatoma regions, we proposed a method to generate optimal laser automatically scanning trajectories inside the areas and between them. The trajectories are tacked using an image-based control scheme. The proposed method and materials were validated experimentally using the lab-made robotic platform. The obtained results in terms of accuracy and behaviour meet the laser surgery requirements perfectly.

This paper deals with an OCT (optical coherence tomography) geometric calibration method. OCT medical imaging system has received a growing interest during the last two decades. In medical purposes, OCT images are generally called optical biopsies which allows in-vivo investigation almost similar to a histopathological study. The physician can rely on the OCT images to establish a rapid and direct diagnosis. But the OCT images formation suffered numerous distortions due in particular to the optical path, from the source to the viewed sample passing through the two reflecting mirrors and a scan objective. The obtained optical biopsies include several spectral and geometric distortions. The proposed calibration model aims to compensate the geometrical ones. More precisely, two models were developed allowing the correction of both 2D images (B-Scan slices) and 3D images (volume). These models were experimentally validated (in both artificial and biological samples) using a spectral domain OCT system. It has demonstrated a significant enhancement of the OCT images accuracy.

Positioning of micro-nanoobjects inside a scanning electron microscope (SEM) for manipulation is a key and challenging task to perform. Often it is performed by skilled operators via teleoperation, which is tedious and lacks repeatability. In this paper, rendering this task as an image-guided problem, we present a frequency domain scheme for automatic control of positioning platform movements. The designed controller uses the relative global image motion computed using the frequency spectral information of the images as visual signal and can provide control up to five degrees of freedom. The proposed approach is validated in simulations as well as experimentally using a high-resolution piezo-positioning platform mounted inside a SEM vacuum chamber. The obtained results quantify the performance of the proposed nanopositioning scheme.

The Remote Centre of Motion (RCM) is an essential movement for most of the medical robotic systems during the minimal invasive surgeries. In literature, there are many references which had analysed the techniques for modelling the problem of RCM constraint. However, only very few have discussed the RCM with Path Following Controller which ensures the complexity of surgical tasks. Therefore, this article is focusing on presenting the Task Priority Controller. Such task controller deploys exteroceptive sensor (visual servoing scheme) in order that the surgical tool follows the reference path while maintaining the RCM constraints. The experimental results showed good performance of the proposed controller.

This paper presents preliminary results on OCT (Optical Coherence Tomography) based control of a concentric tube robot (CTR). The potential use of OCT system is presented then its ability to perform an optical biopsy and an eye-to-hand configuration for robot control is highlighted. A prototype of CTR with three curved tubes is shortly described then a model-less OCT-based controller is detailed. The extraction of visual features from OCT images is performed by normalized cross-correlation. The proposed control scheme was experimentally validated. The positioning accuracy of CTR control was evaluated in benchtop trials. The error in tip placement can reach 10 μm along X and 74 μm along Y according to controller parameters. The 2D positioning (XY plan) is also tested then a final error about 55 μm is achieved.

International audience

This paper deals with the development of a new 6 degrees-of-freedom (DOF) vision-based controller for robot-assisted medical applications. The main objective is to use visual information extracted from Ultrasounds (US) images to control an US probe held by a robot. Instead of the conventional use of visual features e.g., geometric features (points, lines, moments, etc.) in the visual control law design, the described method uses the shearlet coefficients. More precisely, the time-variation of the coarsest level of shearlet decomposition coefficients are linked to the US probe (respectively to the robot) spatial velocity and then related to the task-function control law. The proposed control law was experimentally tested and validated using a realistic abdominal phantom. The obtained results demonstrated promising performances in terms of accuracy, repeatability, robustness and convergence behavior.

This paper proposes the design of a vision-guided control law for microrobotic-assisted biomedical applications. More precisely, the developed control law is to servo an optical coherence tomography (OCT) system in real-time to carry out repetitive optical biopsy tasks. The OCT images are simultaneously used to perform the optical biopsy and to control the sample holder (microrobotic work-flow). Instead of extracting visual features from the OCT images, the vision-based controller uses the concept of frequency domain information to compute relative motion between two successive OCT B-scan (cross-section) images. Therefore, the visual controller is designed to minimize the error between current and desired images by controlling the microrobotic platform. The proposed approach was experimentally validated, demonstrating more than satisfactory results especially in terms of accuracy, convergence, and robustness.

This paper presents the development of a new visual servoing control law using global information image. It consists in the use of the wavelet spatial coefficients as the visual signal to design visual controller. More precisely, we introduce the concept of multiple resolution interaction matrix that links the time-variation of wavelet coefficients to the robot spatial velocity and the associated task function controller. The proposed controller was successfully validated using a 6 degrees-of-freedom 3PPSR parallel kinematic manipulator in an eye-to-hand configuration. Experimental results show the accuracy, efficiency and show good behavior of such features.

This paper deals with the development of a vision-based controller for robot-assisted medical applications. It concerns the use of shearlet coefficients in case of ultrasounds (US) images as visual signal inputs and the design of the associated interaction matrix. The proposed controller was validated in both simulation and on an experimental test bench which consists of a robotic arm holding an US probe in contact with a realistic abdominal phantom. Also, the proposed control scheme was compared to the photometry-based visual servoing approach in order to evaluate its efficiency in different conditions of use (nominal and unfavorable conditions).

Recently, high resolution visual methods based on direct-phase measurement of periodic patterns has been proposed with successful applications to microrobotics. This paper proposes a new implementation of direct-phase measurement methods to achieve 3-DoF (degrees of freedom) visual servoing. The proposed algorithm relies on a single frequency tracking rather than a complete 2D discrete Fourier transform that was required in previous works. The method does not require any calibration step and has many advantages such as high subpixelic resolution, high robustness and short computation time. Several experimental validations (in favorable and unfavorable conditions of use) were performed using a XYθ microrobotic platform. The obtained results demonstrate the efficiency of the frequency-based controller, this in term of accuracy (micrometric error), convergence rate (30 iterations in nominal conditions) and robustness.

This paper addresses the design of a partitioned vision-guided scheme for repetitive optical biopsies. More precisely, our approach uses two image modalities to perform 6 degrees of freedom (DOF) positioning task. The development aims to partition the control into 3 DOF controlled by the B-scan images acquired with an optical coherence tomography (OCT) system and the remaining 3 DOF controlled by the white light images provided by a CCD camera. Moreover, for the control and instead of conventional visual features (e.g., points, lines, moments, etc.) extracted using algorithms combined with visual tracking approaches, our visual servoing method uses the multiresolution wavelet coefficients. The developed method was experimentally validated using a parallel kinematic structure equipped with a Telesto-II OCT benchtop. The validation task consisted of an automatic spatial repositioning of the robotic structure to precisely retrieve the position of an initial optical biopsy. Several tests are achieved, which clearly demonstrate the reliability of the proposed controller.

This paper deals with mutual information-based numerical and physical registration of white light images vs. fluorescence images for microrobotic laser microphonosurgery of the vocal folds. More precisely, it presents two techniques: a numerical registration of multimodal images and a vision feedback control for positioning an endoscope with regards to a preoperative image (fluorescence image). Nelder-Mead Simplex for nonlinear optimization is used to minimize the cost-function. The proposed methods are successfully validated in an experimental set-up using preoperative fluorescence images and real-time white light images of the vocal folds.

Microassembly represents a very promising solution to microproducts and complex Micro-Electro-Mechanical Systems (MEMS) fabrication. Since, in the case of teleoperated assembly, an operator is the main source of errors, there is a great interest in microassembly automation. Its main issue consists in precise estimation of object position. Previous studies demonstrate the possibility of application of modelbased visual tracking algorithms from ViSP (Visual Servoing Platform) library. However, the methods of macroassembly cannot be directly applied when working with microobjects. The characterization of single-view visual tracking notably revealed the complexity of depth estimation in microscale, which is due to small depth variation in the seen images compared with the distance from camera. So, an algorithm of Z coordinate reconstruction using a second camera was developed and analyzed for visual servoing task. It was then used to automate microassembly. Experiments demonstrate the possibility of complex microcomponent automatic microassembly with precision better than 10 micrometers.

Laser surgery requires accurate following of a path defined by the surgeon, while the velocity on this path is dependent on the laser-tissue interaction. Therefore, path following and velocity profile control must be decoupled. In this paper, nonholonomic control of the unicycle model is used to implement velocity-independent visual path following for laser surgery. The proposed controller was tested in simulation, as well as experimentally in several conditions of use: different initial velocities (step input, successive step inputs, sinusoidal inputs), optimized/nonoptimized gains, time-varying path (simulating a patient breathing), and complex curves with curvatures. Thereby, experiments at 587 Hz (frames/s) show an average accuracy lower than 0.22 pixels (~10 μm) with a standard deviation of 0.55 pixels (~25 μm) path following, and a relative velocity distortion of less than 10-6%.

This paper presents two visual servoing approaches for nanopositioning in a scanning electron microscope (SEM). The first approach uses the total pixel intensities of an image as visual measurements for designing the control law. The positioning error and the platform control are directly linked with the intensity variations. The second approach is a frequency domain method that uses Fourier transform to compute the relative motion between images. In this case, the control law is designed to minimize the error i.e. the 2D motion between current and desired images by controlling the positioning platform movement. Both methods are validated at different experimental conditions for a task of positioning silicon microparts using a piezo-positioning platform. The obtained results demonstrate the efficiency and robustness of the developed methods.

This paper deals with the study of a weakly calibrated multiview visual servoing control law for microrobotic laser phonomicrosurgery of the vocal folds. It consists of the development of an endoluminal surgery system for laser ablation and resection of cancerous tissues. More specifically, this paper focuses on the part concerning the control of the laser spot displacement during surgical interventions. To perform this, a visual control law based on trifocal geometry is designed using two cameras and a laser source (virtual camera). The method is validated on a realistic testbench and the straight point-to-point trajectories are demonstrated.

Fast and reliable autofocusing methods are essential for performing automatic nano-objects positioning tasks using a scanning electron microscope (SEM). So far in the literature, various autofocusing algorithms have been proposed utilizing a sharpness measure to compute the best focus. Most of them are based on iterative search approaches; applying the sharpness function over the total range of focus to find an image in-focus. In this paper, a new, fast and direct method of autofocusing has been presented based on the idea of traditional visual servoing to control the focus step using an adaptive gain. The visual control law is validated using a normalized variance sharpness function. The obtained experimental results demonstrate the performance of the proposed autofocusing method in terms of accuracy, speed and robustness.

Despite a large number of proofs of concept in nanotechnologies (e.g., nanosensors), nanoelectromechanical systems (NEMS) hardly come to the market. One of the bottlenecks is the packaging of NEMS which require handling, positioning, assembling and joining strategies in the mesoscale (from 100 nm to 10$\ \mu{\rm m}$, between nanoscale and microscale). It requires models of the interaction forces and adhesion forces dedicated to this particular scale. This paper presents several characteristics of the mesoscale in comparison with nanoscale and microscale. First, it is shown that the distributions of charges observed on the micro-objects and meso-objects would have negligible effects on the nano-objects. Second, the impact of both chemical functionalization and physical nanostructuration on adhesion are presented. Third, the van der Waals forces are increased by local deformations on the mesoscale contrary to the nanoscale where the deformation is negligible. This paper shows some typical characteristics of the mesoscale.

This paper demonstrates accurate micropositioning scheme based on a direct visual servoing process. This technique uses only the pure image signal (photometric information) to design the control law. With respect to traditional visual servoing approaches that use geometric visual features (points, lines ...), the visual features used in the control law are the pixel intensity. The proposed approach was tested in term of accuracy and robustness in several experimental conditions. The obtained results have demonstrated a good behavior of the control law and very good positioning accuracy. The obtained accuracies are estimated to 14 nm, 89 nm, and 0.001 degrees in the x, y and θ axes of positioning platform, respectively.

This paper describes the vision-based methods developed for assembly of complex and solid 3D MEMS (micro electromechanical systems) structures. The microassembly process is based on sequential robotic operations such as planar positioning, gripping, orientation in space and insertion tasks. Each of these microassembly tasks is performed using a pose-based visual control. To be able to control the microassembly process, a 3D model-based tracker is used. This tracker is able to directly provide the 3D micro-object pose at real-time and from only a single view of the scene. The methods proposed in this paper are validated on an automatic assembly of fives silicon microparts of 400 ¿m × 400 ¿m × 100 ¿m on 3-levels. The insertion tolerance (mechanical play) is estimated to 3 ¿m. The precision of this insertion tolerance allows us to obtain solid and complex micro electromechanical structures without any external joining (glue, wending. Promising positioning and orientation accuracies are obtained who can reach 0.3 ¿m in position and 0.2° in orientation.