Safe and intuitive telemanipulation of multiple microrobots is limited by visual occlusions that compromise control and task success. This paper introduces a shared control framework that guarantees safety and visibility by formulating the control problem as a constrained optimization problem. Our framework combines Control Lyapunov Functions (CLFs) for operator-driven stability with High-Order Control Barrier Functions (HOCBFs) to enforce collision and occlusion avoidance, all resolved within a real-time Quadratic Program (QP). A key innovation is a rendering technique based on the QP’s Lagrange multipliers, which are used to decompose the optimal control solution. This method isolates the effects of individual constraints to provide the human operator with distinct, interpretable visuohaptic cues for navigation and avoidance. The architecture also includes a viewpoint optimization system with a virtual autonomous camera to maximize task visibility. The framework is validated via two user studies in simulation and a demonstration on a real-world electromagnetic microrobotics actuation system. Results confirm that our visuo-haptic shared control strategies significantly improve user performance and task completion rates, while the control design provably prevents the microrobots from colliding or entering occluded regions.

Microrobotics implies actuation-related constraints that make safe telemanipulation particularly challenging. We present a haptic shared control system for electromagnetic-based telemanipulation of a pair of microrobots using a constrained optimization framework. Our contributions include: (1) a Quadratic Programming formulation with Control Lyapunov Functions and Control Barrier Functions, for safe and stable navigation in cluttered environments; (2) a shared control architecture, combining a haptic interface and simulation environment, to teleoperate the microrobots and enable micromanipulation capabilities; and (3) haptic shared control strategies offering visuo-haptic cues for task execution. The approach is validated through a user study, highlighting better navigation accuracy, control stability and task efficiency.

The precise manipulation of microrobots presents challenges arising from their small size and susceptibility to external disturbances. To address these challenges, we present the experimental evaluation of a haptic shared control teleoperation framework for the locomotion of multiple microrobots, relying on a kinesthetic haptic interface and a custom electromagnetic system. Six combinations of haptic and shared control strategies are evaluated during a safe 3D navigation scenario in a cluttered environment. 18 participants are asked to steer two spherical magnetic microrobots among obstacles to reach a predefined goal, under different conditions. For each condition, participants are provided with different obstacle avoidance and navigation guidance cues. Results show that providing assistance in avoiding obstacles guarantees safer performance, regardless if the assistance is autonomous or delivered through a haptic repulsive force. Moreover, autonomous obstacle avoidance also reduces the completion time by 30% compared to haptic obstacle avoidance and no obstacle avoidance cases, although haptic feedback is preferred by the users. Finally, providing haptic guidance towards the target improves by the 65% the positioning accuracy of the microrobots with respect to not providing this guidance. We also present some illustrative scenarios to generalize the presented haptic shared control strategies to arbitrary formations of N microrobots, while showing the effectiveness of the method for a clinical use-case of endovascular navigation in simulated environment. Note to Practitioners—The recent increasing interest in microrobotics arises from its potential applications in fields like medicine, manufacturing, and environmental monitoring, enabling highly precise control of minimally invasive tools. By enabling users to teleoperate microscale tools with partial autonomous support, these systems facilitate safe access to confined spaces, enhance task efficiency, and enable navigation in otherwise inaccessible environments. Our presented solution serves as an experimental platform to evaluate the efficacy of different combinations of tactile feedback and partial autonomy during safe navigation tasks, with potential applications spanning microsurgery, drug delivery, microscale manufacturing, and environmental remediation. Further practical adaptation of the system will require defining specific application objectives and specifications, along with potential modifications to the actuation system to accommodate environmental constraints of targeted scenarios.

Small untethered soft robots have potential for diverse applications, particularly in constrained spaces where the use of a tethered device would be infeasible. Examples include biomedical applications such as brachytherapy, fine-needle biospy and micro-needle drug delivery. To advance soft robots towards these applications, there is a need to establish methods for tracking and control using clinically-relevant methods. This study demonstrates motion planning and magnetic control of a soft untethered robot, using ultrasound images as feedback. The closed-loop control of the Millipede soft robot is first validated using a camera-based tracker, where the deviation between the planned path and the trajectory of the robot is 1.71 mm. Afterwards, two methods for ultrasound-based tracking capable of estimating the pose of the robot are proposed, a geometric approach and a convolutional neural network (CNN), and their performance is compared using a video camera as ground truth. Following this, the CNN method replaces the camera tracker to estimate the position and orientation of the robot. The closed-loop system using ultrasound images guides the robot through the workspace while avoiding virtual obstacles, and achieves an average tracking error of 1.59 mm and an angle error of 2.24°.

Magnetic actuation is a versatiletechnology, widely applied in medical robotics for noncontact steering of flexible instruments and untethered agents. In this article, we exploit the benefits of this technology by developing a magnetically actuated flexible catheter capable of the controlled ejection and retrieval of an untethered magnetic capsule. The catheter is actuated by the advanced robotics for magnetic manipulation system. The scanning ultrasound is used for 3-D shape reconstruction of the catheter, with a mean error of 0.37 mm. We use a closed-loop position controller to steer the catheter, reporting a mean error of 0.82 mm. We develop a dynamic model of the capsule and use it to predict the trajectory of the projectile. We demonstrate the targeting of the capsule utilizing the null-space of the catheter actuation (mean residual tip displacement of 0.8 mm). Finally, we perform a delivery of a capsule to and retrieval from a target inaccessible for the catheter tip, demonstrating the capability of our instrument to reach challenging locations.

Control of tetherless magnetically actuated helical robots using rotating dipole fields has a wide variety of medical applications. The most promising technique in manipulation of these robots involves a rotating permanent magnet controlled by a robotic manipulator. In this work, we study the open-loop response of helical robots (in viscous fluids characterized by low Reynolds numbers) in the presence of position constraints on the actuating rotating permanent magnet. We first derive a mapping between the space of the manipulator’s joints, the produced magnetic fields in three-dimensional space, and the translational and rotational velocities of the helical robot. Then, we constrain the 3D position of the rotating dipole field and predict the response of the helical robot by controlling its angular velocity using the constrained mapping. We demonstrate open-loop control and gravity compensation of the robot using the angular velocities of the actuating permanent magnet while enforcing constraints on the end-effector position.

Magnetically actuated soft robots have recently been identified for application in medicine, due to their potential to perform minimally invasive exploration of human cavities. Magnetic solutions permit further miniaturization when compared to other actuation techniques, without loss in functionalities. Our long-term goal is to propose a novel actuation method for magnetically actuated soft robots, based on dual-arm collaborative magnetic manipulation. A fundamental step in this direction is to show that this actuation method is capable of controlling up to 8 coincident, independent Degrees of Freedom (DOFs). In present paper, we prove this concept by measuring the independent wrench components on a second pair of static permanent magnets, by means of a high resolution 6-axis load cell. The experiments show dominant activation of the desired DOFs, with mean cross-activation error of the undesired DOFs ranging from 2% to 10%.

Soft materials with embedded magnetic properties can be actuated in a contactless manner for dexterous motion in restricted and unstructured environments. Magnetic soft robots have been demonstrated to be capable of versatile and programmable untethered motion. However, magnetic soft robots reported in literature are typically actuated by utilizing magnetic fields to generate torques that produce deformation. By contrast, this work investigates the utilization of field gradients to produce tethering forces for anchoring soft robots to the working surface, in conjunction with the use of magnetic fields to generate torques for deformation. The methodology applied here uses a six-coil electromagnetic system for field generation. The approach to achieve the magnetic field and gradients desired for soft robot motion is described, along with the restrictions imposed by Maxwell's equations. The design and fabrication of the soft robots is explained together with calculations to assess the capabilities of the actuation system. Proof-of-concept demonstrations of soft robot motion show Hexapede robots with the ability to `walk' untethered on the ceiling of the workspace, working against gravity; and lightweight Worm robots made of thin strips of material are demonstrated to locomote while staying in contact with the ground.

Magnetically-driven screws operating in soft-tissue environments could be used to deploy localized therapy or achieve minimally invasive interventions. In this work, we characterize the closed-loop behavior of magnetic screws in an agar gel tissue phantom using a permanent magnet-based robotic system with an open-configuration. Our closed-loop control strategy capitalizes on an analytical calculation of the swimming speed of the screw in viscoelastic fluids and the magnetic point-dipole approximation of magnetic fields. The analytical solution is based on the Stokes/Oldroyd-B equations and its predictions are compared to experimental results at different actuation frequencies of the screw. Our measurements matches the theoretical prediction of the analytical model before the step-out frequency of the screw owing to the linearity of the analytical model. We demonstrate open-loop control in two-dimensional space, and point-to-point closed-loop motion control of the screw (length and diameter of 6 mm and 2 mm, respectively) with maximum positioning error of 1.8 mm.

A rotating two-tailed soft microrobot induces a frequency dependent flow-field in low Reynolds number fluids. We use this flow-field to achieve noncontact manipulation of nonmagnetic microbeads with average diameter of 30 μm in 2-D space. Our noncontact manipulation strategy capitalizes on exerting a rotational magnetic torque on the magnetic dipole of the microrobot. The induced flow-field enables microbeads in the surrounding fluid to orbit the microrobot along a sprocketlike trajectory due to a periodic and asymmetric flow-field caused by the two tails. A hydrodynamic model of the two-tailed microrobot and the orbiting microbeads is developed based on the method of regularized Stokeslets for computing Stokes flows. The relations between the angular velocity of the orbiting microbeads and the rotation frequency of the microrobot, their proximity (p), and tail length ratio of the microrobots are studied theoretically and experimentally. Our simulations and experimental results show that the angular velocity of the orbiting microbeads decreases nearly as | p |-2with the distance to the microrobot and its tail length ratio. We also demonstrate closed-loop control of the microbeads toward target positions along sprocketlike trajectories with an average position error of 23.1 ± 9.1 μm (n = 10), and show the ability to swim away without affecting the positioning accuracy after manipulation.

Catheterization instruments are increasingly being improved to accurately diagnose and treat cardiovascular conditions. However, current catheter systems provide limited information about the shape of the catheter and tissue-instrument interaction forces during an intervention. Furthermore, relying on inconsistent feedback of such interaction forces during an intervention may result in tissue injury. This paper presents the first steps to estimate the interaction forces between a catheter and a mock-up arterial environment. We base the proposed method on a Pseudo-Rigid Body approximation of the catheter and integrate three-dimensional shape information provided by Fiber Bragg Grating sensors inside the catheter. The reconstructed forces along the catheter body can be fed back to the surgeon in visual and/or haptic form. In this work, the estimated forces are displayed in real-time in a graphical user interface with the reconstructed catheter shape. Experimental validation demonstrates a root mean square error of 0.03 N and a mean reconstruction error of 0.02 N.

Independent control of microrobots is a cardinal challenge for manipulation at micro/nano scale. In this paper, we design and assemble an electromagnetic setup to overcome some of the major obstacles in the independent control of microrobots. The demanding magnetic requirements are met by the presented experimental testbed that is able to produce magnetic fields and gradients of, respectively, 160 mT and 3.6 T/m at the center of the workspace. Through the design process of this testbed, we analyze the importance of design parameters and derive a quantitative analysis of the requirements for the dissipation of the generated heat. Further, we present and develop the model and software infrastructure, capable of running at 25 Hz, necessary for independent control of multiple microrobots. We also introduce two novel techniques for current-minimizing mapping of the desired forces into currents at the electromagnet. Finally, the capabilities of the setup are demonstrated through independent control of two, both identical and nonidentical, soft-magnetic microspheres in three-dimensional space—with average root mean square errors of 102$\mu$m and peak velocities of up to 331$\mu$m/s.

Minimally invasive surgery involves using flexible medical instruments such as endoscopes and catheters. Magnetically actuated catheters can provide improved steering precision over conventional catheters. However, besides the actuation method, an accurate tip position is required for precise control of the medical instruments. In this study, the tip position obtained from transverse 2D ultrasound images and multicore optical shape sensors are combined using a robust sensor fusion algorithm. The tip position is tracked in the ultrasound images using a template-based tracker and a convolutional neural network based tracker, respectively. Experimental results for a rhombus path are presented, where data obtained from both tracking sources are fused using Luenberger and Kalman state estimators. The mean and standard deviation of the Euclidean error for the Luenberger observer is 0.2 ± 0.11 [mm] whereas for the Kalman filter it is 0.18 ± 0.13 [mm], respectively.

Pneumatic stepper motors are one of the promising alternative actuation methods for motion control in environments where electromagnetic (EM) motors cannot be used. Due to the lack of commercial off-the-shelf products, researchers working on MR compatible robotics have to develop their own pneumatic actuators. This imposes extensive costs and delays on the development process. Additionally, the current solutions are limited in their range of specifications and are difficult to manufacture. In this paper, proof-of-concept-prototypes for a family of parametrically designed, electromagnetically stealth, rotational pneumatic stepper motors are presented. The main objective of the paper is to demonstrate a general purpose non-electromagnetic actuation method, which can be customized and integrated into any design. Customizability, miniaturization, safety and affordability are some of the key features of the presented work. The developed prototypes are entirely 3D-printed and contain no sealing, bearing or lubrication. Thanks to the low production cost, the motor can be used as a disposable part in surgical applications. Experiments demonstrate effectiveness of the design in terms of cost-efficiency, versatility, MRI-compatibility, speed and performance. In order to optimize the design and control algorithm, empirical equations are presented describing response time of a pneumatic system to sequential pressure signals. A rotational speed of 800 rpm, total volume of 4.6 cm3and resolution of 3° are some of the design attributes. The effects of clearance on stick-slip effect and leakage in a 3D printed cylinder-piston are also presented.

Minimally invasive surgery can benefit greatly from utilizing micro-agents. These miniaturized agents need to be clearly visualized and precisely controlled to ensure the success of the surgery. Since medical imaging modalities suffer from low acquisition rate, multi-rate sampling methods can be used to estimate the intersample states of micro-agents. Hence, the sampling rate of the controller can be virtually increased even if the position data is acquired using a slow medical imaging modality. This study presents multi-rate Luenberger and Kalman state estimators for visual tracking of micro-agents. The micro-agents are tracked using sum of squared differences and normalized cross correlation based visual tracking. Further, the outputs of the two methods are merged to minimize the tracking error and prevent tracking failures. During the experiments, the micro-agents with different geometrical shapes and sizes are imaged using a 2D ultrasound machine and a microscope, and manipulated using electromagnetic coils. The multi-rate state estimation accuracy is measured using a high speed camera. The precision of the tracking and multi-rate state estimation are verified experimentally under challenging conditions. For this purpose, an elliptical shaped magnetic micro-agent with a length of 48 pixels is used. Maximum absolute error in x and y axes are 2.273 and 2.432 pixels for an 8-fold increase of the sample rate (25 frames per second), respectively. During the experiments, it was observed that the micro-agents could be tracked more reliably using normalized cross correlation based visual tracking and inters ample states could be estimated more accurately using Kalman state estimator. Experimental results show that the proposed method could be used to track micro-agents in medical imaging modalities and estimate system states at intermediate time instants in real-time.

Untethered miniature robotics have recently shown promising results in several scenarios at the microscale, such as targeted drug delivery, microassembly, and biopsy procedures. However, the vast majority of these small-scale robots have very limited manipulation capabilities, and none of the steering systems currently available enables humans to intuitively and effectively control dexterous miniaturized robots in a remote environment. In this paper, we present an innovative micro-teleoperation system with haptic assistance for the intuitive steering and control of miniaturized self-folding soft magnetic grippers in 2-D space. The soft grippers can be wirelessly positioned using weak magnetic fields and opened/closed by changing their temperature. An image-guided algorithm tracks the position of the controlled miniaturized gripper in the remote environment. A haptic interface provides the human operator with compelling haptic sensations about the interaction between the gripper and the environment as well as enables the operator to intuitively control the target position and grasping configuration of the gripper. Finally, magnetic and thermal control systems regulate the position and grasping configuration of the gripper. The viability of the proposed approach is demonstrated through two experiments involving 26 human subjects. Providing haptic stimuli elicited statistically significant improvements in the performance of the considered navigation and micro-manipulation tasks.

The use of magnetic resonance (MR) images for needle-based interventions offers several advantages over other types of imaging modalities (e.g., high tissue contrast and no radiation). However, MR-guided interventions face challenges related to electromagnetic compatibility of medical devices and real-time tracking of surgical instruments. This work presents a flexible needle steering system that combines an MR-compatible robot and a Fiber Bragg Grating (FBG)-based needle tip tracker. The MR images are used to localize obstacles and targets, while the FBG sensors provide strain measurements for online estimation of the needle tip position. A pre-operative planner defines the needle entry point and desired path, while a model predictive controller calculates the needle rotation during the insertion. To the best of the authors knowledge, this is the first work that fuses MR images and FBG-based tracking to steer a flexible needle in closed-loop inside the MR bore. The system is validated by steering a bevel-tipped flexible needle towards a physical target in gelatin phantoms and biological tissues. The needle reaches the target in all trials with an average targeting error of 2.76 mm. Disregarding the target displacement during the insertion, the average targeting error drops to 1.74 mm. The preliminary results demonstrate the feasibility of combining MR images and FBG-based needle tip tracking to steer a flexible needle in clinical procedures. In order to move towards to a clinically-relevant application, the design of a flexible Nitinol biopsy needle is also presented and evaluated by experiments in a prostate of a bull. The flexible needle presented a curvature 2.5 times larger than a conventional biopsy needle while maintaining the ability to collect tissue samples.

Soft miniaturized untethered grippers can be used to manipulate and transport biological material in unstructured and tortuous environments. Previous studies on control of soft miniaturized grippers employed cameras and optical images as a feedback modality. However, the use of cameras might be unsuitable for localizing miniaturized agents that navigate within the human body. In this paper, we demonstrate the wireless magnetic motion control and planning of soft untethered grippers using feedback extracted from B-mode ultrasound images. Results show that our system employing ultrasound images can be used to control the miniaturized grippers with an average tracking error of 0.4±0.13 mm without payload and 0.36±0.05 mm when the agent performs a transportation task with a payload. The proposed ultrasound feedback magnetic control system demonstrates the ability to control miniaturized grippers in situations where visual feedback cannot be provided via cameras.

Magnetic interaction can be utilized for remote actuation of surgical manipulators. However, platforms currently available for that purpose have limited workspaces, inadequate field strength or very low bandwidth of the electrical subsystem. In this paper, we present BigMag, a novel platform capable of magnetic steering of continuum manipulators for medical purposes. BigMag comprises of 6 mobile coils and is capable of generating the fields of at least 40 mT in any direction at every point of its workspace. Moreover, we introduce a mathematical model for 3D mobile coil arrays. Each coil is modelled using finite element data adjusted by measurement-based correction, (a maximum observed mean error between the model and the prediction of 3.36 ± 5.62%). The model for a full system is validated in two tasks. In the first task, the system executes a prescribed rotating field (mean error between the model and measurement of 7.51% and minimum R2of 0.964). The second task tests the estimation of the field for known 3D trajectories (minimum R2of 0.967). The investigation concludes with a demonstration of BigMag capabilities in actuation of magnetic catheters in confined spaces usinguser-controlled steering.

The presence of force feedback in medical instruments has been proven to reduce tissue damage. In order to provide force feedback, information about the interaction forces between the instrument and the environment must be known. Direct measurement of these forces by commercial sensors is not feasible due to space limitations. Thus, in this study we propose to estimate the interaction forces using strain measurements from Fiber Bragg Grating (FBG) sensors. These measurements can also be used for shape sensing and as a result both force and shape can be sensed simultaneously. For force sensing two models are proposed and compared. The first is based on a Rigid Link approximation, while the second uses the Cosserat rod theory. The models are validated experimentally using a tendon-driven continuum manipulator that is subjected to forces at the tip. The force estimates from the models are compared to the measurements from a commercial force sensor. Mean absolute errors of 11.2 mN (6.9%) and 15.9 mN (8.3%) are observed for the Rigid Link model and Cosserat model, respectively.

Needle insertion procedures are commonly performed in current clinical practice for diagnostic and therapeutic purposes. Although prevailing technology allows accurate localization of lesions, they cannot yet be precisely targeted. Needle steering is a promising technique to overcome this challenge. In this paper, we describe the development of a novel steering system for an actuated-tip flexible needle. Strain measurements from an array of Fiber Bragg Grating (FBG) sensors are used for online reconstruction of the needle shape in 3D-space. FBG-sensor data is then fused with ultrasound images obtained from a clinically-approved Automated Breast Volume Scanner (ABVS) using an unscented Kalman filter. A new ultrasound-based tracking algorithm is developed for the robust tracking of the needle in biological tissue. Two experimental cases are presented to evaluate the proposed steering system. In the first case, the needle shape is reconstructed using the tracked tip position in ultrasound images and FBG-sensor measurements, separately. The reconstructed shape is then compared with the actual 3D needle shape obtained from the ABVS. In the second case, two steering experiments are performed to evaluate the overall system by fusing the FBG-sensor data and ultrasound images. Average targeting errors are 1.29±0.41 mm and 1.42±0.72 mm in gelatin phantom and biological tissue, respectively.

Micro-sized agents can benefit robotic minimally invasive surgery since they can be inserted into the human body and use natural pathways such as arteries and veins or the gastrointestinal tract, to reach their target for drug delivery or diagnosis. Recently, miniaturized agents with shape-changing and gripping capabilities have provided significant advantages in performing grasping, transportation, and manipulation tasks. In order to robustly perform such tasks, it is of utmost importance to properly estimate their overall configuration. This paper presents a novel solution to the problem of estimating and tracking the 3D position, orientation and configuration of the tips of miniaturized grippers from RGB marker-less visual observations obtained by a microscope. We consider this as an optimization problem, seeking for the gripper model parameters that minimize the discrepancy between hypothesized instances of the gripper model and actual observations of the miniaturized gripper. This optimization problem is solved using a variant of the Particle Swarm Optimization algorithm. The proposed approach has been evaluated on several image sequences showing the grippers moving, rotating, opening/closing and grasping biological material.

Accurate closed-loop control of continuum manipulators requires integration of both models that describe their motion and methods to evaluate manipulator shape. This work presents a model that approximates the continuous shape of a continuum manipulator by a serial chain of rigid links, connected by flexible rotational joints. This rigid-link model permits a description of manipulator shape under different loading conditions. A kinematic controller, based on the manipulator Jacobian of the proposed rigid-link model, is implemented and realizes trajectory tracking, while using the kinematic redundancy of the manipulator to perform a secondary task of avoiding obstacles. The controller is evaluated on an experimental testbed, consisting of a planar tendon-driven continuum manipulator with two bending segments. Fiber Bragg grating (FBG) sensors are used to reconstruct 3-D manipulator shape, and is used as feedback for closed-loop control of the manipulator. Manipulator steering is evaluated for two cases: the first case involving steering around a static obstacle and the second case involving steering along a straight path while avoiding a moving obstacle. Mean trajectory tracking errors are 0.24 and 0.09 mm with maximum errors of 1.37 and 0.52 mm for the first and second cases, respectively. Finally, we demonstrate the possibility of FBG sensors to measure interaction forces, while simultaneously using them for shape sensing.

Needle insertions are common during surgical procedures. Accurately delivering the needle at a specific location in the human body is of importance for the clinical outcome of the procedure. Studies have already shown that robotically inserting traditional needles with a bevel tip can improve targeting accuracy. However, steering of such needles requires spinning the needle, which may lead to additional tissue damage. Therefore, we propose a novel design consisting of a flexible needle with a tendon-driven actuated-tip. Changing the orientation of the actuated-tip allows to control the steering direction of the needle and the amount of deflection. We derive the kinematic model which describes the needle path given the actuated-tip orientation based on nonholonomic kinematics. We present a method for steering the needle towards a target location in soft tissue. This method incorporates online parameter estimation in order to adapt for changes in tissue stiffness. Needle insertion experiments are performed in soft-tissue simulants, made from porcine gelatin. Needle tip pose is measured during insertion using Fiber Bragg Grating (FBG) based shape reconstruction. Results show that the needle can be steered towards targets located at 20 mm from the initial insertion axis, at a depth of 100 mm with a mean targeting error of 2.02 mm.

In targeted therapy, clusters of drug carriers (nanoparticles and microparticles) could be in contact with a surface such as the lumen of blood vessels and the interior of the gastrointestinal tract. We study the motion characteristics of clusters of microparticles when they slide on a surface under the influence of weak oscillating magnetic fields (less than 11 mT). The oscillating magnetic fields exert a magnetic torque on the microparticles and allow them to oscillate, and hence overcome the static friction and slide on a surface. We characterize the frequency response of clusters of microparticles by applying oscillating magnetic fields with a frequency range of 0 Hz to 55 Hz, in the presence of a constant magnetic field gradient (0.9 T/m). Clusters of 3 to 4 and 5 to 9 microparticles achieve maximum sliding speeds of 1100 µm/s and 1150 µm/s, at oscillating magnetic fields of 30 Hz. In addition, we experimentally demonstrate closed-loop motion control of the clusters with maximum position error of 20 µm. Furthermore, we show that the magnetic field gradient required to drive a cluster of microparticles (with 3 to 4 microparticles) decreases by 75% in the presence of oscillating magnetic fields from 5 Hz to 50 Hz.

In contact micromanipulation, the adhesive forces between manipulators and microobjects decrease the chances of achieving successful releases at the desired positions. We study a non-contact micromanipulation technique of microbeads (300 μm in average diameter) using clusters of paramagnetic microparticles (100 μm in average diameter). This non-contact micromanipulation is done using the hydrodynamic forces instead of the interaction forces in contact manipulation, and hence eliminates the adhesive forces that decrease the chances of achieving successful releases. Motion of the cluster of microparticles results in a pressure gradient (within the vicinity of the microbead in a fluid) that derives and steers the microbeads without contact. The microparticles are moved under the influence of controlled magnetic field gradient to push or pull the microbeads towards reference positions. We achieve non-contact manipulation via pushing and pulling at average speeds of 219 μm/s and 258 μm/s for the microbead, respectively (using cluster of 10 microparticles). The noncontact pushing and pulling localize the microbeads within the vicinity of reference positions with average steady-state errors of 177 μm and 100 μm, respectively. Moreover, we experimentally demonstrate non-contact microassembly of 3 microbeads into an L-shape at a task completion time of 25 seconds.

We control the motion of helical microrobots with average diameter of 500 μm in two-dimensional (2D) and three-dimensional (3D) spaces using two synchronized rotating dipole fields. The utilization of the two synchronized dipole fields not only increases the magnetic torque exerted on the magnetic dipole of the helical microrobot but also eliminates the magnetic field gradients along its lateral directions. Our finite element simulations and experimental results show that the utilization of two rotating dipole fields increases the magnetic field by 100%, as opposed to single rotating magnetic field. In addition, we show that the magnetic field gradient within the workspace of the microrobot is eliminated. Therefore, the lateral oscillations of the helical microrobot are mitigated within the center of two rotating dipole fields, and hence the motion of the microrobot is stabilized inside tubes with relatively large inner diameters, as opposed to the diameter of the helical microrobot. This strategy allows the microrobot to compensate for gravity and swim in 3D space inside water reservoirs at an average speed of 0.25 body lengths per second. In addition, closed-loop motion control of the helical microrobot is achieved in 2D space at an average speed of 2 mm/s and maximum steady-state error of 100 μm.

Flexible bevel-tipped needle steering has been an active research topic since the last decade. However, most of the work presented so far performs flexible needle steering in static phantoms or non-static virtual environments. The insertion of flexible bevel-tipped needles into soft-tissue subjected to motion disturbances is still an open research problem. In this paper, we propose a flexible needle steering algorithm that is able to handle physiological motion disturbances. The system estimates the disturbance using the force information provided by a force sensor placed in contact with the soft-tissue. The system is evaluated through experiments steering a flexible needle towards a physical target into a moving soft-tissue phantom. Three experimental cases with motion disturbances based on clinical procedures in the lung, kidney and heart are used to assess the targeting error. The average targeting error of all 15 experimental trials is 1:05±0:41mm. Our results demonstrate the ability of the proposed system to compensate for motion disturbances applied to the soft-tissue phantom.

Controlling the motion of microrobots based on feedback provided using an imaging modality is essential to make them clinically viable. In this study, we demonstrate the wireless magnetic-based motion control of paramagnetic microparticles using ultrasound feedback. This control is accomplished by pulling the microparticles using the magnetic field gradients towards the reference position through feedback provided by an ultrasound system. First, position of the microparticles is determined using the ultrasound images. Second, calibration of the ultrasound-based tracking of microparticles is achieved and verified using a calibrated microscopic system. Third, the feedback provided by the ultrasound system is used in the implementation of a proportional-derivative magnetic-based control system. This control system allows us to achieve point-to-point control of microparticles with an average position tracking error of 48±59 μm, whereas a control system based on a microscopic system achieves an average position tracking error of 21±26 μm. The positioning accuracy accomplished using our ultrasound magnetic-based control system demonstrates the ability to control microrobotic systems in situations where visual feedback cannot be provided via microscopic systems.

Percutaneous needle insertions are commonly performed for diagnostic and therapeutic purposes. Accurate placement of the needle tip is important to the success of many needle procedures. The current needle steering systems depend on needle-tissue-specific data, such as maximum curvature, that is unavailable prior to an interventional procedure. In this paper, we present a novel three-dimensional adaptive steering method for flexible bevel-tipped needles that is capable of performing accurate tip placement without previous knowledge about needle curvature. The method steers the needle by integrating duty-cycled needle steering, online curvature estimation, ultrasound-based needle tracking, and sampling-based motion planning. The needle curvature estimation is performed online and used to adapt the path and duty cycling. We evaluated the method using experiments in a homogenous gelatin phantom, a two-layer gelatin phantom, and a biological tissue phantom composed of a gelatin layer and in vitro chicken tissue. In all experiments, virtual obstacles and targets move in order to represent the disturbances that might occur due to tissue deformation and physiological processes. The average targeting error using our new adaptive method is 40% lower than using the conventional non-adaptive duty-cycled needle steering method.

Paramagnetic microparticles moving through fluids have the potential to be used in many applications, including microassembly, micromanipulation, and highly localized delivery of therapeutic agents inside the human body. Paramagnetic microparticles with diameters of approximately 100 μm can be wirelessly controlled by externally applying magnetic field gradients using electromagnets. In this paper, we introduce a motion planner to guide a spherical paramagnetic microparticle to a target while avoiding obstacles. The motion planner explicitly considers uncertainty in the microparticle's motion and maximizes the probability that the microparticle avoids obstacle collisions and reaches the target. To enable effective consideration of uncertainty, we use an Expectation Maximization (EM) algorithm to learn a stochastic model of the uncertainty in microparticle motion and state sensing from experiments conducted in a 3D 8-electromagnet microparticle testbed. We apply the motion planner in a simulated 3D environment with static obstacles and demonstrate that the computed plans are more likely to result in task success than plans based on traditional metrics such as shortest path or maximum clearance.

Recent technological advancements in cardiovascular surgery such as transapical transcatheter aortic valve implantation (TA-TAVI) enabled treatment to elderly that were initially declined surgery. However, valve malpositioning during TA-TAVI have been reported in several cases. In this preliminary study, we present a novel approach in which a Robotically-Actuated Delivery Sheath (RADS) is used to potentially facilitate valve positioning. A model is developed that describes the shape and articulating tip position of the RADS. We developed a two-dimensional ultrasound tracking method that evaluates the tip position of the RADS in ultrasound images. Both modeling and ultrasound tracking are combined into an integrated system that facilitates closed-loop control of the articulating tip of the RADS. Experiments are performed in order to evaluate the tracking accuracy of the RADS. Experiments show mean positioning errors of approximately 2 mm along the x- and y-axes. Our study demonstrates that the RADS can potentially provide compensation for beating heart and respiratory motions during valve positioning and deployment in TA-TAVI.

We experimentally demonstrate that using oscillating weak magnetic fields a sperm-shaped microrobot (which we refer to as MagnetoSperm) can swim using flagellar propulsion and slide on a surface under water. The sperm morphology allows the MagnetoSperm to mimic the locomotion mechanism of a living sperm cell. The MagnetoSperm is designed and developed with a magnetic head and a flexible tail to provide a magnetic dipole moment and propulsion, respectively. The head oscillates under the influence of controlled oscillating weak magnetic fields (~5 mT). This oscillation generates a thrust force in the flexible tail, and hence allows the MagnetoSperm to overcome the drag and friction forces during swimming and sliding on a surface, respectively. Point-to-point open- and closed-loop control of the MagnetoSperm are accomplished using an electromagnetic system under microscopic guidance. This motion control is done in two cases, i.e., swimming in water and sliding on a surface. At oscillating magnetic field of 5 Hz and 45 Hz, the MagnetoSperm swims at an average swimming speed of 32 μm/s (0.1 body lengths per second) and 158 μm/s (0.5 body lengths per second), respectively. At the same frequencies, the MagnetoSperm slides on the bottom of a petri-dish at an average speed of 21 μm/s (0.07 body lengths per second) and 6 μm/s (0.02 body lengths per second), respectively.

Needle insertion in soft-tissue is a minimally invasive surgical procedure which demands high accuracy. In this respect, robotic systems with autonomous control algorithms have been exploited as the main tool to achieve high accuracy and reliability. However, for reasons of safety and acceptance by the surgical community, autonomous robotic control is not desirable. Thus, it is necessary to focus more on techniques enabling clinicians to directly control the motion of surgical tools. In this work we address that challenge and present a novel teleoperated robotic system able to steer flexible needles. The proposed system tracks the position of the needle using an ultrasound imaging system, and, from that, it computes needle's ideal position and orientation to reach a given target. The master haptic interface then provides mixed kinesthetic-vibratory navigation cues about this ideal position and orientation to the clinician as she steers the needle. Six subjects carried out an experiment of teleoperated needle insertion into a soft-tissue phantom. They showed a mean targeting error of 1.36 mm. An additional experiment of remote teleoperation has been carried out to highlight the passivity-based stability of the proposed system.

We demonstrate the closed-loop motion control of self-propelled microjets inside a fluidic microchannel. The motion control of the microjets is achieved in hydrogen peroxide solution with time-varying flow rates, under the influence of the controlled magnetic fields and the self-propulsion force. Magnetic dipole moment of the microjets is characterized using the U-turn and the rotating field techniques. The characterized magnetic dipole moment has an average of$\hbox{1.4}\times \hbox{10}^{-13}$A.m$^{2}$at magnetic field, linear velocity, and boundary frequency of 2 mT, 100$\mu$m/s, and 25 rad/s, respectively. We implement a closed-loop control system that is based on the characterized magnetic dipole moment of the microjets. This closed-loop control system positions the microjets by directing the magnetic field lines toward the reference position. Experiments are done using a magnetic system and a fluidic microchannel with a width of 500$\mu$m. In the absence of a fluid flow, our control system positions the microjets at an average velocity and within an average region-of-convergence (ROC) of 119$\mu$m/s and 390$\mu$m, respectively. As a representative case, we observe that our control system positions the microjets at an average velocity and within an average ROC of 90$\mu$m/s and 600$\mu$m and 120$\mu$m/s and 600$\mu$m when a flow rate of 2.5$\mu$l/min is applied against and along the direction of the microjets, respectively. Furthermore, the average velocity and ROC are determined throughout the flow range (0 to 7.5$\mu$l/min) to characterize the motion of the microjets inside the microchannel.

Needle insertion procedures are commonly used for surgical interventions. In this paper, we develop a three-dimensional (3D) closed-loop control algorithm to robotically steer flexible needles with an asymmetric tip towards a target in a soft-tissue phantom. Twelve Fiber Bragg Grating (FBG) sensors are embedded on the needle shaft. FBG sensors measure the strain applied on the needle during insertion. A method is developed to reconstruct the needle shape using the strain data obtained from the FBG sensors. Four experimental cases are conducted to validate the reconstruction method (single-bend, double-bend, 3D double-bend and drilling insertions). In the experiments, the needle is inserted 120 mm into a soft-tissue phantom. Camera images are used as a reference for the reconstruction experiments. The results show that the mean needle tip accuracy of the reconstruction method is 1.8 mm. The reconstructed needle shape is used as feedback for the steering algorithm. The steering algorithm estimates the region that the needle can reach during insertion, and controls the needle to keep the target in this region. Steering experiments are performed for 110 mm insertion, and the mean targeting accuracy is 1.3 mm. The results demonstrate the capability of using FBG sensors to robotically steer needles.

We use a cluster of paramagnetic microparticles to carry out a wireless two-dimensional microassembly operation. A magnetic-based manipulation system is used to control the motion of the cluster under the influence of the applied magnetic fields. Wireless motion control of the cluster is implemented at an average velocity and maximum position tracking error of 144 μm/s and 50 μm, respectively. This control is used to achieve point-to-point positioning of the cluster, manipulation of microobjects, and assembly of microobjects into a microstructure. The control system achieves stable positioning of the cluster, while simultaneously compensating for the planar drag forces on the cluster and the microobject. The presented magnetic-based microassembly technique allows for the selective pushing and pulling of microobjects with specific geometries towards their destinations inside a microstructure in an execution time of 18 s, within a workspace of 1.8 mm × 2.4 mm.

We demonstrate the closed-loop control of a magnetotactic bacterium (MTB), i.e., Magnetospirillum magnetotacticum, within a micro-fabricated maze using a magnetic-based manipulation system. The effect of the channel wall on the motion of the MTB is experimentally analyzed. This analysis is done by comparing the characteristics of the transient- and steady-states of the controlled MTB inside and outside a micro-fabricated maze. In this analysis, the magnetic dipole moment of our MTB is characterized using a motile technique (the u-turn technique), then used in the realization of a closed-loop control system. This control system allows the MTB to reach reference positions within a micro-fabricated maze with a channel width of 10 μm, at a velocity of 8 μm/s. Further, the control system positions the MTB within a region-of-convergence of 10 μm in diameter. Due to the effect of the channel wall, we observe that the velocity and the positioning accuracy of the MTB are decreased and increased by 71% and 44%, respectively.

Needle insertion is one of the most commonly performed minimally invasive procedures. Visualization of the needle during insertion is key for either successful diagnosis or therapy. This work presents the real-time three-dimensional tracking of flexible needles during insertion into a soft-tissue simulant using a two-dimensional (2D) ultrasound transducer. The transducer is placed perpendicular to the needle tip to measure its position. During insertion the transducer is robotically repositioned to track the needle tip. Positioning of the transducer is accomplished by a compensator, that uses the needle insertion velocity corrected by needle tip velocities to determine out-of-plane motion. Experiments are performed to validate the needle tip pose during tracking. The maximum mean errors in needle tip position along the x-, y- and z-axes are 0.64 mm, 0.25 mm and 0.27 mm, respectively. The error in tip orientations (θ-about the y-axis and φ-about the z-axis) are 2.68° and 2.83°, respectively. This study demonstrates the ability to compute the needle tip pose using a 2D ultrasound transducer. The tip pose can be used to robotically steer needles, and thereby improve accuracy of medical procedures.

Magnetic drug carriers such as microrobots and paramagnetic microparticles have the potential to increase the therapeutic indices by selectively targeting the diseased tissue. These magnetic microobjects can be controlled using magnetic-based manipulation systems. In this study, we analyze a minimum input motion control to minimize the currents at each of the electromagnets of a magnetic system. This minimum input control allows us to achieve point-to-point closed-loop motion control of microparticles in the three-dimensional space, at an average speed of 198 μm/s, and maximum root mean square position tracking error of 104 μm. The minimum input control system is further evaluated by comparing norm-2 of its resulting current vector to the current vector of a proportional-integral (PI) control system. This comparison shows that the minimum input control achieves 11% decrease in the current input, as opposed to the PI control system. However, the PI control system achieves 43% and 285% higher average speed and positioning accuracy, respectively, as opposed to the minimum input controller. The magnetic-based minimum input control can be used to perform closed-loop control of magnetic microrobots while decreasing the current input.

We provide a comparative study between two self-propelled microrobots, i.e., magnetotactic bacteria and microjets. This study includes characterization of their fluidic properties (linear and rotational drag coefficients) based on their morphologies and characterization of their magnetic properties using the rotating-field technique. Further, the control characteristics of our microrobots are evaluated in the transient- and steady-states. The average boundary frequencies of our magnetotactic bacteria and microjets are 2.2 rad/s and 25.1 rad/s, respectively. The characterized fluidic properties and boundary frequencies are used in the characterization of the magnetic properties of our microrobots. The average magnetic dipole moments of our magnetotactic bacteria and microjets are 1.4×10-17A.m2and 1.5×10-13A.m2at magnetic field of 2 mT and linear velocities of 32 μm/s (approximately 6 body lengths per second) and 119 μm/s (approximately 2 body lengths per second), respectively. These characterized magnetic dipole moments are utilized in the realization of closed-loop control systems for the magnetotactic bacteria and microjets. Our closed-loop control system positions the magnetotactic bacteria and the microjets within the vicinity of reference positions with average diameters of 23 μm (approximately 4 body lengths) and 417 μm (approximately 8 body lengths), respectively.

Flexible minimally invasive surgical instruments can be used to target difficult-to-reach locations within the human body. Accurately steering these instruments requires information about the three-dimensional shape of the instrument. In the current study, we use an array of Fiber Bragg Grating (FBG) sensors to reconstruct the shape of a flexible instrument. FBG sensors have several advantages over existing imaging modalities, which makes them well-suited for use in a clinical environment. An experimental testbed is presented in this study, which includes a tendon-driven manipulator. A nitinol FBG-wire is fabricated, on which an array of twelve FBG sensors are integrated, and distributed over four different sets. This wire is positioned in the backbone of the manipulator. Axial strains are measured using the FBG sensors, from which the curvature of the manipulator is calculated. The three-dimensional manipulator shape is reconstructed from the curvature, which is used to steer the manipulator tip. We are able to steer the manipulator along various trajectories (two-dimensional and three-dimensional), and also reject disturbance loads. We observe a minimum mean tracking error of 0.67 mm for the circular trajectory in closed-loop control. This study demonstrates the potential of steering flexible minimally invasive surgical instruments using an array of FBG sensors.

Needle insertion procedures are commonly used for diagnostic and therapeutic purposes. In this paper, an image-guided control system is developed to robotically steer flexible needles with an asymmetric tip. Knowledge about needle deflection is required for accurate steering. Two different models to predict needle deflection are presented. The first is a kinematics-based model, and the second model predicts needle deflection that is based on the mechanics of needle-tissue interaction. Both models predict deflection of needles that undergo multiple bends. The maximum targeting errors of the kinematics-based and the mechanics-based models for 110-mm insertion distance using a φ 0.5-mm needle are 0.8 and 1.7 mm, respectively. The kinematics-based model is used in the proposed image-guided control system. The control system accounts for target motion during the insertion procedure by detecting the target position in each image frame. Five experimental cases are presented to validate the real-time control system using both camera and ultrasound images as feedback. The experimental results show that the targeting errors of camera and ultrasound image-guided steering toward a moving target are 0.35 and 0.42 mm, respectively. The targeting accuracy of the algorithm is sufficient to reach the smallest lesions (φ 2 mm) that can be detected using the state-of-the-art ultrasound imaging systems.

In this paper two different approaches to guarantee stability of bilateral telemanipulation systems are discussed. Both approaches inject damping into the system to guarantee passivity of the interaction with the device in the presence of time delays in the communication channel. The first approach derives tuning rules for a fixed viscous damper, whereas the second approach employs modulated dampers based upon the measured energy exchange with the device and enforces passivity in the time domain. Furthermore, a theoretical minimum damping injection scheme is sketched that shows that the fixed damping approach is inherently conservative with respect to guaranteeing stability. Experimental results show that both the theoretical minimum damping scheme and a time domain passivity algorithm are successful in stabilizing the telemanipulation system for large time delays with lower gains of the damping elements than derived by the fixed damping injection approach. However, as damping is inherently present in the system, the fixed damping tuning rules can be used to identify if a time domain passivity algorithm is needed given boundary conditions on the actual time delays.

A system is developed that can reconstruct the pose of flexible endoscopic instruments that are used in advanced flexible endoscopes using solely the endoscopic images. Four markers are placed on the instrument, whose positions are measured in the image. These measurements are compared to a three-dimensional rendered model of the instrument. The pseudo-inverse of the interaction matrix between the state of the model and the marker positions in the image is used to update the state such that the model will track the real instrument. An experiment was performed in which the instrument was moved inside a colon model, while the tip position was simultaneously measured with an electromagnetic tracking system. The root mean square errors of the position estimation were 2.3 mm, 2.2 mm and 1.7 mm in the horizontal (x), vertical (y) and away-from-camera (z) directions, respectively.

This video shows a system that estimates the pose of a flexible endoscopic instrument, based on the endoscopic images. A three-dimensional rendering of the instrument is matched to the actual instrument that is observed through the endoscopic camera. This system was evaluated in an anatomical model of a colon. The estimated position of the tip of the instrument was compared to measurements performed with an electromagnetic tracker. The errors of the position estimation were 2 mm, 2.2 mm and 1.7 mm in the horizontal (x), vertical (y) and away-from-camera (z) directions, respectively.

This work investigates the utilization of microparticles for the wireless sensing of interaction forces in magnetic-based manipulation systems. The proposed force estimation approach allows for using microparticles in sensing the interaction forces at hard-to-reach regions to avoid the mechanical and electronic complexities associated with physical force sensors. Based on the velocity of the microparticle and the applied currents at each of the electromagnets of the magnetic system, an interaction force observer is designed to estimate the contact forces between the microparticle and a soft-tissue simulant with different elasticities. Experimentally, a magnetic system is utilized to steer a microparticle towards a soft-tissue simulant to carry out force sensing. The experimental results show that forces in the range of nano-Newton can be estimated without nano-force sensors. The estimated interaction forces due to this contact can be used either in sensing and diagnosis applications, or in the realization of a force control system.

This work presents an energy-based state estimation formalism for a class of dynamical systems with inaccessible/unknown outputs, and systems at which sensor utilization is impractical, or when measurements can not be taken. The power-conserving physical interconnections among most of the dynamical subsystems allow for power exchange through their power ports. Power exchange is conceptually considered as information exchange among the dynamical subsystems and further utilized to develop a natural feedback-like information from a class of dynamical systems with inaccessible/unknown outputs. This information is used in the design of an energy-based state observer. Convergence stability of the estimation error for the proposed state observer is proved for systems with linear dynamics. Furthermore, robustness of the convergence stability is analyzed over a range of parameter deviation and model uncertainties. Experiments are conducted on a dynamical system with a single input and multiple inaccessible outputs (Fig. 1) to demonstrate the validity of the proposed energy-based state estimation formalism.

Passivity of bilateral telemanipulation systems ensures stability of the interaction with such systems. In the frequency domain, passivity of a linear time invariant approximation of the system can be designed for a considered set of operating conditions. Non-linear control structures have been proposed that enforce passivity of the system in the time domain. In this paper, extensions are proposed that increase the complimentarity of the frequency- and time domain approaches. The combination of both approaches allows a guaranteed measure of transparency to be designed in the frequency domain for a desired set of operating conditions. For operating conditions outside the desired set, stable interaction is guaranteed by the non-linear passivity enforcing control structure. Simulation results of the combined approach are presented that show that the stability properties of the bilateral controller designed in the frequency domain are improved and the transparency properties are improved with respect to those of the standard passivity-enforcing algorithm in the time domain.

This paper describes the design of a system for controlling the position of spherical paramagnetic microparticles that have an average diameter of 100 µm. The focus of this study lies in designing and implementing a system that uses microscopic images and electromagnets. Preliminary experiments have been done to verify the feasibility of the system to track and control the position of these particles. A vibrating sample magnetometer was used to determine the magnetic moment of the particles. Finite element method simulations were used to verify the magnetic behavior of the designed setup. The system was used to position the particles within 8.4µm of a setpoint, achieving speeds of up to 235µm s−1. We also demonstrated that the particle could follow a circular and a figure-eight path.

The goal of this study is to improve the efficiency and efficacy of the standard colonoscopy procedure. This is done by addressing the intuitiveness of colonoscope control. For this purpose an interface in the form of a grip was designed that allows the user to intuitively steer and drive the colonoscope. The Grip controls the orientation of the tip as if the colonoscope were a stiff instrument that pivots at the anus of a patient. To test the principle, experiments were conducted on a simulator operated by novice subjects. Initial experiments show a significant decrease in introduction time of 156 seconds (p<0.005). This technology will enhance current colonoscopy practice and open up possibilities for future applications of colonoscopy.

A position and orientation sensing system is developed for the feedback control of endoscopic instruments in advanced flexible endoscopes. The images that are taken by the endoscopic camera are used to match a kinematic model to the observed instrument. Using the pseudo-inverse of the Jacobian of the forward kinematics, the estimated state of the model is continuously updated so as to match feature points from the images to the model. An experiment was performed inside a colon model, in which reference markers with known locations were touched with the instrument. The root mean square position estimation errors were 1.7 mm, 1.2 mm and 3.6 mm in the horizontal (x), vertical (y), and away-from-camera (z) directions, respectively.

When a needle is inserted into soft tissue, interaction forces are developed at the needle tip and along the needle shaft. The needle tip force is due to cutting of the tissue, and the force along the needle shaft is due to friction between needle and tissue. In this study, the friction force is determined for needles inserted into a gelatine phantom at insertion velocities of 10 mm/s and 20 mm/s. The friction force is found to be dependent on the insertion velocity. The needle tip force is calculated using the friction and insertion force, and is used as input for a mechanics-based model which predicts the amount of needle deflection. In the model, the needle is considered to be a cantilever beam supported by springs which have needle-tissue interaction stiffness (Ke). The value of the interaction stiffness is evaluated by comparing results from experiments and simulation. A mechanical needle insertion device is used to insert needles. Needle deflection during insertion is determined using a needle tip tracking algorithm. Results of this study provide insight into the mechanics of needle-tissue interaction, and can be used in studies for robotically steering needles into soft tissue.

In this paper, a two-layer approach is presented to guarantee the stable behavior of bilateral telemanipulation systems in the presence of time-varying destabilizing factors such as hard contacts, relaxed user grasps, stiff control settings, and/or communication delays. The approach splits the control architecture into two separate layers. The hierarchical top layer is used to implement a strategy that addresses the desired transparency, and the lower layer ensures that no “virtual” energy is generated. This means that any bilateral controller can be implemented in a passive manner. Separate communication channels connect the layers at the slave and master sides so that information related to exchanged energy is completely separated from information about the desired behavior. Furthermore, the proposed implementation does not depend on any type of assumption about the time delay in the communication channel. By complete separation of the properties of passivity and transparency, each layer can accommodate any number of different implementations that allow for almost independent optimization. Experimental results are presented, which highlight the benefit of the proposed framework.

Passivity of virtual environments running in discrete time is a sufficient condition for stability of the system. The framework for passive sampled Port-Hamiltonian systems allows multi-dimensional virtual environments exhibiting internal dynamic behavior to be computed on a discrete medium in a passive manner. It is shown that a causality analysis is required in the framework to detect if any of the model elements have, a time dependent change of energy function in the energy balance of the system. The Standard Linear Solid model, which is often used to simulate the visco-elastic interaction with soft biological tissue is used as an example. Simulated and experimental results are provided to demonstrate the benefit of the described framework. It is shown that using this approach a multi-dimensional model which is passive in the continuous domain remains passive in the discrete domain, whereas a standard discretization approach can become non-passive.

In bilateral telemanipulation algorithms based on time-domain passivity, internal friction in the devices poses an additional energy drain. Based on a model of the friction, the dissipated energy can be estimated and reclaimed inside the energy balance of the control algorithm. As long as the estimate is conservative, passivity of the entire system is maintained. In this paper we consider two types of friction and discuss the influence of two types of measurement noise. Without noise compensation the dissipated energy is largely overestimated. A compensation method based on the probability density of the noise is proposed. This leads to an energy estimate which is always conservative even in the presence of measurement noise and does not require additional filtering. Simulation results are provided that show the increase in obtained transparency when this energy compensation technique is applied.

Manually steering the tip of a flexible endoscope to navigate through an endoluminal path relies on the physician's dexterity and experience. In this paper we present the realization of a robotic flexible endoscope steering system that uses the endoscopic images to control the tip orientation towards the direction of the lumen. Two image-based control algorithms are investigated, one is based on the optical flow and the other is based on the image intensity. Both are evaluated using simulations in which the endoscope was steered through the lumen. The RMS distance to the lumen center was less than 25% of the lumen width. An experimental setup was built using a standard flexible endoscope, and the image-based control algorithms were used to actuate the wheels of the endoscope for tip steering. Experiments were conducted in an anatomical model to simulate gastroscopy. The image intensity-based algorithm was capable of steering the endoscope tip through an endoluminal path from the mouth to the duodenum accurately. Compared to manual control, the robotically steered endoscope performed 68% better in terms of keeping the lumen centered in the image.

The asymmetry of a bevel-tip needle results in the needle naturally bending when it is inserted into soft tissue. In this study we present a mechanics-based model that calculates the deflection of the needle embedded in an elastic medium. Microscopic observations for several needle-gel interactions were used to characterize the interactions at the bevel tip and along the needle shaft. The model design was guided by microscopic observations of several needle-gel interactions. The energy-based model formulation incorporates tissue-specific parameters such as rupture toughness, nonlinear material elasticity, and interaction stiffness, and needle geometric and material properties. Simulation results follow similar trends (deflection and radius of curvature) to those observed in macroscopic experimental studies of a robot-driven needle interacting with different kinds of gels. These results contribute to a mechanics-based model of robotic needle steering, extending previous work on kinematic models.