Abstract In this manuscript, we present a novel cutting method that involves using a vertex‐snapping strategy to fit the boundary surface onto the cutting path while avoiding generating new elements. We employ a point cloud with polynomial fitting to generate the cutting path, allowing for operation with unscheduled cuts and potential perturbations. Efficient geometry operations are developed to handle topological changes during progressive cutting. While it is challenging to optimize the mesh quality and accurately align the cut surface with the cutting path, we propose an innovative strategy that converts this geometric problem into a quasi‐static elastic problem. This involves solving a constrained elastic problem within an auxiliary simulation, where the system optimizes the mesh quality when reaching equilibrium. Furthermore, we propose modifications to a GPU‐based matrix‐free solver, enabling efficient updates of the precomputed data stored in the GPU memory and thus ensuring real‐time performance.

ABSTRACT This article presents a fluoroscopy image‐based registration method along with a comprehensive protocol for robotic needle insertion in radiofrequency ablation (RFA) to treat liver cancer. The proposed method uses real‐time fluoroscopic images acquired from a C‐ARM system and integrates an inverse finite element (FE) simulation to compute robotic commands for accurate and adaptive needle steering. The registration procedure is fully automated and involves the injection of multiple radiopaque markers into the liver, enabling precise anatomical registration and targeted tumor localization. A key challenge addressed in this work is the integration of this image‐based registration with the inverse biomechanical simulation used to guide the robot during insertion. We describe how registration constraints can be mapped onto the surface of the biomechanical model to ensure consistent alignment between image data and robotic actuation. Designed to be adaptable to varying levels of radiologist expertise and applicable across a wide range of tumor locations, this method provides a robust and versatile solution for improving the accuracy and safety of minimally invasive liver cancer treatments.

This paper presents an innovative approach for executing robotic needle insertion within deformable living organs. The objective is to maintain the insertion pivot point on the skin, which remains stationary. At the same time, the organs undergo displacement and deformation due to respiration. Therefore, real-time control and precise needle steering are crucial. The proposed method relies on isolated objective constraints to ensure the objectives while steering the needle along a predefined trajectory. The needle insertion process benefits from Finite Element (FE) models to simulate the environment and address the inverse problem to drive the robot’s end effector (EE) by re-evaluating the objective functions in the constraint space for each time step. So, the desired motion of the robot’s EE could be calculated at a small cost for non-linear functions in real-time, resulting in better precision and reducing stress caused to the organs.

Needle-based procedures such as biopsies or radiofrequency ablation (RFA) of tumors are often considered to diagnose and treat liver cancer for their low invasiveness but raise difficulties for practitioners related to needle placement and visibility of internal anatomical structures. Efforts are being conducted to build real-time needle insertion simulators with both visual and haptic rendering, facing challenges related to model accuracy and real-time computational performance. This work focuses on the contact model involved in needle-tissue interactions in order to improve the realism of the resulting haptic rendering. We present a novel method to update the compliant coupling at high rates of a complete contact system involving the mechanics of a large object and the complete model of a flexible needle. These updates allow to adapt the contact directions to the needle deformations in the haptic thread, with the aim of improving the resulting haptic feedback. Updates of contact directions and the related mechanical system according to high-rate deformations decrease force feedback artifacts associated with low-rate mechanics while maintaining high-rate performances for the haptic loop.

Abstract This paper presents a fast method to compute large‐scale problems in real‐time finite element simulations in the presence of contact and friction. The approach uses a precondition‐based contact resolution that performs a Cholesky decomposition at low frequency. On exploiting the sparsity in assembled matrices, we propose a reduced and parallel computation scheme to address the expensive computation of the Schur‐complement arisen by detailed mesh and accurate contact response. An efficient GPU‐based solver is developed to parallelise the computation, making it possible to provide real‐time simulations in the presence of coupled constraints for contact and friction response. In addition, the pre‐conditioner is updated at low frequency, implying reuse of the factorised system. To benefit a further speedup, we propose a strategy to share the resolution information between consecutive time steps. We evaluate the performance of our method in different contact applications and compare it with typical approaches on CPU and GPU.

This paper deals with the problem of needle steering in deformable tissues subject to physiological motions. A novel shared control method is proposed, which combines an automatic needle steering algorithm with the motions applied by the radiologist, in order to place the needle tip at the desired location. The core motivation is to leave potentially dangerous decisions and actions to the practitioner, whereas complex non-intuitive manipulations of the needle are performed automatically, in particular to compensate for breathing motions. The most original part of the present work lies in the method used to combine user inputs with a closed-loop automatic needle steering control method based on inverse Finite Element simulations. The method is evaluated with a realistic virtual environment using 2D X-ray projection images. The results are compared with those obtained with a fully teleoperated system, on the one hand, and with a fully automatic solution, on the other hand. These experiments show that the shared control solution allows for a better needle tip placement when only projection imaging is available.

This paper introduces a method for robotic steering of a flexible needle inside moving and deformable tissues. The method relies on a set of objective functions allowing to automatically steer the needle along a predefined path. In order to follow the desired trajectory, an inverse problem linking the motion of the robot end effector with the objective functions is solved using a Finite Element simulation. The main contribution of the article is the new constraint-based formulation of the objective functions allowing to: 1) significantly reduce the computation time; 2) increase the accuracy and stability of the simulation-guided needle insertion. The method is illustrated, and its performances are characterized in a realistic framework, using a direct simulation of the respiratory motion generated from in vivo data of a pig. Despite the highly non-linear behavior of the numerical simulation and the significant deformations occurring during the insertion, the obtained performances enable the possibility to follow the trajectory with the desired accuracy for medical purpose.

This paper introduces a new approach for the control of a robotic system interacting with deformable structures. The method is applied to needle insertion procedures, which are among the least invasive surgical approaches to access deep internal structures with sometimes poor access conditions. Yet, during the insertion both tissues and needles deform resulting in a displacement of targets identified at the planning step and significantly raising the technical difficulty of these approaches. Robotic assistance may offer new possibilities to enforce the accuracy of the needle's positioning, but the deformation of tissues remains an open problem. In this paper, we propose a numerical approach where finite-element (FE) models are used in a close-control robotic loop. We introduce a complete forward simulation of deformable structures (needle and environment) and constraint-based interaction models allowing for the simulation of needle insertion and complex nonlinear phenomena (friction, puncture, and insertion) at a high frequency. For the control, we numerically derive the so-called Jacobian of the Simulation using an inverse method. The most original aspect of this paper lies in the fact that inverse steps are performed in constraints space, allowing this way for fast estimation of the Jacobian (i.e., between 40 and 100 Hz). The method is validated both numerically and experimentally using a flexible needle inserted inside a deformable foam. We show that the robot is able to follow a given trajectory, defined during the planning step, taking into account any occurring deformation of both the needle and the foam during the insertion; without any need for tracking the needle neither the target nor the trajectory.

This paper introduces an Augmented Reality (AR) system for open liver surgery. Although open surgery remains the gold-standard for the treatment of complex tumors and central lesions, technological issues actually prevent using AR with sufficient accuracy for clinical use. We propose a markers-based method allowing for the tracking and the deformation of a preoperative model in real-time during the surgery. Markers are manually placed on the surface of the organ after opening the abdominal cavity, and tracked in real-time by a set of infrared cameras. Our framework is composed of both a nonrigid initial registration method, providing an estimation of the location of the markers in the preoperative model, and a realtime tracking algorithm to deform the model during the surgery (even for large deformation or partial occlusion of the organ). The method is validated on both synthetic and ex-vivo samples; in addition, we demonstrate its applicability in the operating room during a liver resection surgery on a human patient. Preliminary studies provided promising results to improve the location of tumors, and to help surgeons into planning the ideal resection intraoperatively.

In this paper we introduce a method for semiautomatic registration of 3D deformable models using 2D shape outlines (silhouettes) extracted from a monocular camera view. Our framework is based on the combination of a biomechanical model of the organ with a set of projective constraints influencing the deformation of the model. To enforce convergence towards a global minimum for this ill-posed problem we interactively provide a rough (rigid) estimation of the pose. We show that our approach allows for the estimation of the non-rigid 3D pose while relying only on 2D information. The method is evaluated experimentally on a soft silicone gel model of a liver, as well as on real surgical data, providing augmented reality of the liver and the kidney using a monocular laparoscopic camera. Results show that the final elastic registration can be obtained in just a few seconds, thus remaining compatible with clinical constraints. We also evaluate the sensitivity of our approach according to both the initial alignment of the model and the silhouette length and shape.

This paper introduces a new method for automatic robotic needle steering in deformable tissues. The main contribution relies on the use of an inverse Finite Element (FE) simulation to control an articulated robot interacting with deformable structures. In this work we consider a flexible needle, embedded in the end effector of a 6 arm Mitsubishi RV1A robot, and its insertion into a silicone phantom. Given a trajectory on the rest configuration of the silicone phantom, our method provides in real-time the displacements of the articulated robot which guarantee the permanence of the needle within the predefined path, taking into account any undergoing deformation on both the needle and the trajectory itself. A forward simulation combines i) a kinematic model of the robot, ii) FE models of the needle and phantom gel iii) an interaction model allowing the simulation of friction and puncture force. A Newton-type method is then used to provide the displacement of the robot to minimize the distance between the needle's tip and the desired trajectory. We validate our approach with a simulation in which a virtual robot can successfully perform the insertion while both the needle and the trajectory undergo significant deformations.

This paper presents an original method for interactions' haptic rendering when treating hyperelastic materials. Such simulations are known to be difficult due to the non-linear behavior of hyperelastic bodies; furthermore, haptic constraints enjoin contact forces to be refreshed at least at 1000 updates per second. To enforce the stability of simulations of generic objects of any range of stiffness, this method relies on implicit time integration. Soft tissues dynamics is simulated in real time (20 to 100 Hz) using the Multiplicative Jacobian Energy Decomposition (MJED) method. An asynchronous preconditioner, updated at low rates (1 to 10 Hz), is used to obtain a close approximation of the mechanical coupling of interactions. Finally, the contact problem is linearized and, using a specific-loop, it is updated at typical haptic rates (around 1000 Hz) allowing this way new simulations of prompt stiff-contacts and providing a continuous haptic feedback as well.

We introduce a new method for simulating frictional contact between volumetric objects using interpenetration volume constraints. When applied to complex geometries, our formulation results in dramatically simpler systems of equations than those of traditional mesh contact models. Contact between highly detailed meshes can be simplified to a single unilateral constraint equation, or accurately processed at arbitrary geometry-independent resolution with simultaneous sticking and sliding across contact patches. We exploit fast GPU methods for computing layered depth images, which provides us with the intersection volumes and gradients necessary to formulate the contact equations as linear complementarity problems. Straightforward and popular numerical methods, such as projected Gauss-Seidel, can be used to solve the system. We demonstrate our method in a number of scenarios and present results involving both rigid and deformable objects at interactive rates.

The Gauss-Seidel method is very efficient for solving problems such as tightly-coupled constraints with possible redundancies. However, the underlying algorithm is inherently sequential. Previous works have exploited sparsity in the system matrix to extract parallelism. In this paper, we propose to study several parallelization schemes for fully-coupled systems, unable to be parallelized by existing methods, taking advantage of recent many-cores architectures offering fast synchronization primitives. Experimental results on both multi-core CPUs and recent GPUs show that our proposed method is able to fully exploit the available units, whereas trivial parallel algorithms often fail. This method is illustrated by an application in medical intervention planning, where it is used to solve a linear complementary problem (LCP) expressing the contacts applied to a deformable body.