Intraocular surgery, one of the most challenging discipline of microsurgery, requires sensory and motor skills at the limits of human physiological capabilities combined with tremendously difficult requirements for accuracy and steadiness. Nowadays, robotics combined with advanced imaging has opened conspicuous and significant directions in advancing the field of intraocular microsurgery. Having patient treatment with greater safety and efficiency as the final goal, similar to other medical applications, robotics has a real potential to fundamentally change microsurgery by combining human strengths with computer and sensor-based technology in an information-driven environment. Still in its early stages, robotic assistance for intraocular microsurgery has been accepted with precaution in the operating room and successfully tested in a limited number of clinical trials. However, owing to its demonstrated capabilities including hand tremor reduction, haptic feedback, steadiness, enhanced dexterity, micrometer-scale accuracy, and others, microsurgery robotics has evolved as a very promising trend in advancing retinal surgery. This paper will analyze the advances in retinal robotic microsurgery, its current drawbacks and limitations, as well as the possible new directions to expand retinal microsurgery to techniques currently beyond human boundaries or infeasible without robotics.

In this paper we describe work towards retinal vessel cannulation using an actively stabilized handheld robot, guided by monocular vision. We employ a previously developed monocular camera based surface reconstruction method using automated laser beam scanning over the retina. We use the reconstructed plane to find a coordinate transform between the 2D image plane coordinate system and the global 3D frame. Within a hemispherical region around the target, we use motion scaling for higher precision. The contribution of this work is the homography matrix estimation using monocular vision and application of the previously developed laser surface reconstruction to Micron guided vein cannulation. Experiments are conducted in a wet eye phantom to show the higher accuracy of the surface reconstruction as compared to standard stereo reconstruction. Further, experiments to show the increased surgical accuracy due to motion scaling are also carried out.

This paper presents robot-aided intraocular laser surgery using a handheld robot known as Micron. The micromanipulator incorporated in Micron enables visual servoing of a laser probe, while maintaining a constant distance of the tool tip from the retinal surface. The comparative study was conducted with various control methods for evaluation of robot-aided intraocular laser surgery.

Vitreoretinal microsurgery requires precise hand-eye coordination to manipulate delicate structures within the eye on the order of tens of microns. To achieve these tasks, surgeons use tools of diameter 0.9 mm or less to access the eye's interior structures. The level of force required during these manipulations is often below the human tactile threshold, requiring the surgeon to rely on subtle visual cues or to apply larger forces above the tactile threshold for feedback. However, both of these methods can lead to tissue damage. Excursions can be made into tissues which are not felt by the surgeon, while larger forces have a higher chance of damaging tissue within the eye. To prevent damage to the retina and other anatomy, we present the implementation of hybrid position/force control operating in the sub-tactile force range for a handheld robotic system. This approach resulted in a 42% reduction in the mean force and 52% reduction in maximum force during peeling tasks.

This paper presents a technique for automated intraocular laser surgery using a handheld micromanipulator known as Micron. The novel handheld manipulator enables the automated scanning of a laser probe within a cylinder of 4 mm long and 4 mm in diameter. For the automation, the surface of the retina is reconstructed using a stereomicroscope, and then preplanned targets are placed on the surface. The laser probe is precisely located on the target via visual servoing of the aiming beam, while maintaining a specific distance above the surface. In addition, the system is capable of tracking the surface of the eye in order to compensate for any eye movement introduced during the operation. We compared the performance of the automated scanning using various control thresholds, in order to find the most effective threshold in terms of accuracy and speed. Given the selected threshold, we conducted the handheld operation above a fixed target surface. The average error and execution time are reduced by 63.6% and 28.5%, respectively, compared to the unaided trials. Finally, the automated laser photocoagulation was demonstrated also in an eye phantom, including compensation for the eye movement.

Computer-aided intraocular surgery requires precise, real-time knowledge of the vasculature during retinal procedures such as laser photocoagulation or vessel cannulation. Because vitreoretinal surgeons manipulate retinal structures on the back of the eye through ports in the sclera, voluntary and involuntary tool motion rotates the eye in the socket and causes movement to the microscope view of the retina. The dynamic nature of the surgical workspace during intraocular surgery makes mapping, tracking, and localizing vasculature in real time a challenge. We present an approach that both maps and localizes retinal vessels by temporally fusing and registering individual-frame vessel detections. On video of porcine and human retina, we demonstrate real-time performance, rapid convergence, and robustness to variable illumination and tool occlusion.

This paper presents the characterization and comparison of physiological tremor for pointing tasks in multiple environments, as a baseline for performance evaluation of microsurgical robotics. Previous studies have examined the characteristics of physiological tremor under laboratory settings as well as different operating conditions. However, different test methods make the comparison of results across trials and conditions difficult. Two vitroretinal microsurgeons were evaluated while performing a pointing task with no entry-point constraint, constrained by an artificial eye model, and constrained by a rabbit eye in vivo. For the three respective conditions the 3D RMS positioning error was 144 μm, 258 μm, and 285 μm, and maximum 3D error was 349 μm, 647 μm, and 696 μm. A spectral analysis was also performed, confirming a distinct peak near in the 6-12 Hz frequency range, characteristic of hand tremor during tasks in all three environments.

Bevel-tipped flexible needles can be robotically steered to reach clinical targets along curvilinear paths in 3D. Manual needle insertion allows the clinician to control the insertion speed, ensuring patient safety. This paper presents a control law for automatic 3D steering of manually inserted flexible needles, enabling path-following control. A look-ahead proportional controller for position and orientation is presented. The look-ahead distance is a linear function of insertion speed. Simulations in a 3D brain-like environment demonstrate the performance of the proposed controller. Experimental results also show the feasibility of this technique in 2D and 3D environments.

This paper describes an inexpensive pico-projector-based augmented reality (AR) display for a surgical microscope. The system is designed for use with Micron, an active handheld surgical tool that cancels hand tremor of surgeons to improve microsurgical accuracy. Using the AR display, virtual cues can be injected into the microscope view to track the movement of the tip of Micron, show the desired position, and indicate the position error. Cues can be used to maintain high performance by helping the surgeon to avoid drifting out of the workspace of the instrument. Also, boundary information such as the view range of the cameras that record surgical procedures can be displayed to tell surgeons the operation area. Furthermore, numerical, textual, or graphical information can be displayed, showing such things as tool tip depth in the work space and on/off status of the canceling function of Micron.

Peeling delicate retinal membranes, which are often less than 5 µm thick, is one of the most challenging retinal surgeries. Preventing rips and tears caused by tremor and excessive force can decrease injury and reduce the need for follow up surgeries. We propose the use of a fully handheld microsurgical robot to suppress tremor while enforcing helpful constraints on the motion of the tool. Using stereo vision and tracking algorithms, the robot activates motion-scaled behavior as the tip reaches the surface, providing finer control during the critical step of engaging the membrane edge. A hard virtual fixture just below the surface limits the total downward force that can be applied. Furthermore, velocity limiting during the peeling helps the surgeon maintain a smooth, constant force while lifting and delaminating the membrane. On a phantom consisting of plastic wrap stretched across a rubber slide, we demonstrate our approach reduces maximum force by 40-70%.

This paper presents the design and analysis of a handheld manipulator for vitreoretinal microsurgery and other biomedical applications. The design involves a parallel micromanipulator utilizing six piezoelectric linear actuators, combining compactness with a large range of motion and relatively high stiffness. Given the available force of the actuators, the overall dimension of the micromanipulator was optimized considering realistic external loads on a remote center of motion representing the point of expected contact with the sclera of the eye during microsurgery. Based on optimization and workspace analysis, a benchtop version of the micromanipulator was built with a base diameter of 25 mm and a height of 50 mm. It provides a hemispherical workspace of 4.0 mm diameter at the tool tip. The manipulation performance of the constructed manipulator was measured under a lateral load applied at the remote center of motion. The micromanipulator tolerated side loads up to 200 mN.

Highly accurate positioning is fundamental to the performance of vitreoretinal microsurgery. Of vitreoretinal procedures, membrane peeling is among the most prone to complications since extremely delicate manipulation of retinal tissue is required. Associated tool-to-tissue interaction forces are usually below the threshold of human perception, and the surgical tools are moved very slowly, within the 0.1-0.5 mm/s range. During the procedure, unintentional tool motion and excessive forces can easily give rise to vision loss or irreversible damage to the retina. A successful surgery includes two key features: controlled tremor-free tool motion and control of applied force. In this study, we present the potential benefits of a micro-force sensing robot in vitreoretinal surgery. Our main contribution is implementing fiber Bragg grating based force sensing in an active tremor canceling handheld micromanipulator, known as Micron, to measure tool-to-tissue interaction forces in real time. Implemented auditory sensory substitution assists in reducing and limiting forces. In order to test the functionality and performance, the force sensing Micron was evaluated in peeling experiments with adhesive bandages and with the inner shell membrane from chicken eggs. Our findings show that the combination of active tremor canceling together with auditory sensory substitution is the most promising aid that keeps peeling forces below 7 mN with a significant reduction in 2-20 Hz oscillations.

This paper presents a framework for localizing a miniature epicardial crawling robot, HeartLander, on the beating heart using only 6-degree-of-freedom position measurements from an electromagnetic position tracker and a dynamic surface model of the heart. Using only this information, motion and observation models of the system are developed such that a particle filter can accurately estimate not only the location of the robot on the surface of the heart, but also the pose of the heart in the world coordinate frame as well as the current physiological phase of the heart. The presented framework is then demonstrated in simulation on a dynamic 3-D model of the human heart and a robot motion model which accurately mimics the behavior of the HeartLander robot.

We describe the design and performance of a hand-held actively stabilized tool to increase accuracy in micro-surgery or other precision manipulation. It removes involuntary motion such as tremor by actuating the tip to counteract the effect of the undesired handle motion. The key components are a three-degree-of-freedom piezoelectric manipulator that has 400 μm range of motion, 1 N force capability, and bandwidth over 100 Hz, and an optical position measurement subsystem that acquires the tool pose with 4 μm resolution at 2000 samples/s. A control system using these components attenuates hand motion by at least 15 dB (a fivefold reduction). By considering the effect of the frequency response of Micron on the human visual feedback loop, we have developed a filter that reduces unintentional motion, yet preserves intuitive eye-hand coordination. We evaluated the effectiveness of Micron by measuring the accuracy of the human/machine system in three simple manipulation tasks. Handheld testing by three eye surgeons and three non-surgeons showed a reduction in position error of between 32% and 52%, depending on the error metric.

Precise movement during micromanipulation becomes difficult in submillimeter workspaces, largely due to the destabilizing influence of tremor. Robotic aid combined with filtering techniques that suppress tremor frequency bands increases performance; however, if knowledge of the operator's goals is available, virtual fixtures have been shown to greatly improve micromanipulator precision. In this paper, we derive a control law for position-based virtual fixtures within the framework of an active handheld micromanipulator, where the fixtures are generated in real-time from microscope video. Additionally, we develop motion scaling behavior centered on virtual fixtures as a simple and direct extension to our formulation. We demonstrate that hard and soft (motion-scaled) virtual fixtures outperform state-of-the-art tremor cancellation performance on a set of artificial but medically relevant tasks: holding, move-and-hold, curve tracing, and volume restriction.

Active compensation of physiological tremor for handheld micromanipulators depends on fast control and actuation responses. Because of real-world latencies, real-time compensation is usually not completely effective at eliminating unwanted hand motion. By modeling tremor, more effective cancellation is possible by anticipating future hand motion. We propose a feedforward control strategy that utilizes tremor velocity from a state-estimating Kalman filter. We demonstrate that estimating hand motion in a feedforward controller overcomes real-world latencies in micromanipulator actuation. In hold-still tasks with a fully handheld micromanipulator, the proposed feedforward approach improves tremor rejection by over 50%.

HeartLander, a small mobile robot designed to provide treatments to the surface of the beating heart, overcomes a major difficulty of minimally invasive cardiac surgery, providing a stable operating platform. This is achieved inherently in the way the robot adheres to and crawls over the surface of the heart. This mode of operation does not require physiological motion compensation to provide this stable environment; however, modeling of physiological motion is advantageous in providing more accurate position estimation as well as synchronization of motion to the physiological cycles. The work presented uses an Extended Kalman Filter framework to estimate parameters of non-stationary Fourier series models of the motion of the heart due to the respiratory and cardiac cycles as well as the position of the robot as it moves over the surface of the heart. The proposed method is demonstrated in the laboratory with HeartLander operating on a physiological motion simulator. Improved performance is demonstrated in comparison to the filtering methods previously used with HeartLander. The use of detected physiological cycle phases to synchronize locomotion of HeartLander is also described.

In microsurgery, a surgeon often deals with anatomical structures of sizes that are close to the limit of the human hand accuracy. Robotic assistants can help to push beyond the current state of practice by integrating imaging and robot-assisted tools. This paper demonstrates control of a handheld tremor reduction micromanipulator with visual servo techniques, aiding the operator by providing three behaviors: snap-to, motion-scaling, and standoff-regulation. A stereo camera setup viewing the workspace under high magnification tracks the tip of the micromanipulator and the desired target object being manipulated. Individual behaviors activate in task-specific situations when the micromanipulator tip is in the vicinity of the target. We show that the snap-to behavior can reach and maintain a position at a target with an accuracy of 17.5 ± 0.4μm Root Mean Squared Error (RMSE) distance between the tip and target. Scaling the operator's motions and preventing unwanted contact with non-target objects also provides a larger margin of safety.

Robotic assistance enhances conventional endoscopy; yet, limitations have hindered its mainstream adoption for cardiac surgery. HeartLander is a miniature mobile robot that addresses several of these limitations by providing precise and stable access over the surface of the beating heart in a less-invasive manner. The robot adheres to the heart and navigates to any desired target in a semiautonomous fashion. The initial therapies considered for HeartLander generally require precise navigation to multiple surface targets for treatment. To balance speed and precision, we decompose any general target acquisition into navigation to the target region followed by fine positioning to each target. In closed-chest, beating-heart animal studies, we demonstrated navigation to targets located around the circumference of the heart, as well as acquisition of target patterns on the anterior and posterior surfaces with an average error of 1.7 mm. The average drift encountered during station-keeping was 0.7 mm. These preclinical results demonstrate the feasibility of precise semiautonomous delivery of therapy to the surface of the beating heart using HeartLander.

We present an investigation into modeling of athetoid motion and prediction of user intent, for use in assistive computer interfaces during icon-clicking tasks. Data were recorded from three athetoid patients during unassisted icon-clicking trials with an isometric joystick. In order to facilitate development and testing of filter designs without the difficulty of repeated testing with human subjects, a quantitative model of the recorded patient data was developed using pseudoinverse methods. Using this model within the visuomotor control loop for the icon-clicking task, a prediction filter was then developed to reduce the target acquisition time. The filter is based on a novel "autoregressive stretching window" model which selects five data points evenly distributed across the input and output histories to predict the intended target, together with a second-order system that smoothes the movement of the cursor. On average, the filter demonstrated a reduction of target acquisition time by a factor of 2.7 in experiments with the patient models.

With the increasing popularity of actuators involving smart materials like piezoelectric, control of such materials becomes important. The existence of the inherent hysteretic behavior hinders the tracking accuracy of the actuators. To make matters worse, the hysteretic behavior changes with rate. One of the suggested ways is to have a feedforward controller to linearize the relationship between the input and output. Thus, the hysteretic behavior of the actuator must be first modeled by sensing the relationship between the input voltage and output displacement. Unfortunately, the hysteretic behavior is dependent on individual actuator and also environmental conditions like temperature. In this fast moving world, time is money and it is very costly to model the hysteresis regularly. In addition, the hysteretic behavior of the actuators also changes with age. Base on the studies done on the phenomena hysteretic behavior with rate, this paper proposes an adaptive rate-dependent feedforward controller with Prandtl-Ishlinskii (PI) hysteresis operators for piezoelectric actuators. This adaptive controller is achieved by adapting the coefficients to manipulate the weights of the play operators. Actual experiments are conducted to demonstrate the effectiveness of the adaptive controller.

Robotic technologies provide new ways to compensate quasi-periodic biological motion, enabling higher surgical accuracy without invasive measures such as cardiopulmonary bypass. This paper describes current research in robotic compensation of hand tremor, respiratory motion, and heartbeat during surgical procedures. An analysis of each physiological motion pattern is provided, as well as a description of novel compensation techniques

This paper develops a physical model of a MEMS capacitive accelerometer in order to use the accelerometer effectively in low-g motion tracking applications. The proposed physical model includes common physical parameters used to rate an accelerometer: scale factor, bias, and misalignment. Simple experiments used to reveal the behavior and characteristics of these parameters are described. A phenomenological modeling method is used to establish mathematical representations of these parameters in relation to errors such as nonlinearity, hysteresis, cross-axis effect, and temperature effect, without requiring a complete understanding of the underlying physics. Experimental results are presented, in which the physical model reduces RMSE by 93.1% in comparison with the manufacturer's recommended method.

This paper presents the theory and modeling of a quaternion-based augmented state Kalman filter for real-time orientation tracking of a handheld microsurgical instrument equipped with a magnetometer-aided all-accelerometer inertial measurement unit (IMU). The onboard sensing system provides two complementary sources of orientation information. The all-accelerometer IMU provides a high resolution but drifting angular velocity estimate, while the magnetic north vector is combined with the estimated gravity vector to yield a non-drifting but noisy orientation estimate. Analysis of the dominant stochastic noise components of the sensors and derivation of the noise covariance are presented. The proposed Kalman filter obtains a non-drifting orientation estimate with improved resolution by incorporating the motion dynamics of the instrument during microsurgery and models the angular velocity drift explicitly as extra dynamic states.

We present the design of an all-accelerometer inertial measurement unit (IMU). The IMU forms part of an intelligent hand-held microsurgical instrument that senses its own motion, distinguishes between hand tremor and intended motion, and compensates in real-time the erroneous motion. The new IMU design consists of three miniature dual-axis accelerometers, two of which are housed in a sensor suite at the distal end of the instrument handle, and one located at the proximal end close to the instrument tip. By taking the difference between the accelerometer readings, we decouple the inertial and gravitational accelerations from the rotation-induced (centripetal and tangential) accelerations, hence simplifies the kinematic computation of angular motions. We have shown that the error variance of the Euler orientation parameters /spl theta//sub x/, /spl theta//sub y/ and /spl theta//sub z/ is inversely proportional to the square of the distance between the three sensor locations. Comparing with a conventional three gyros and three accelerometers IMU, the proposed design reduces the standard deviation of the estimates of translational displacements by 29.3% in each principal axis and those of the Euler orientation parameters /spl theta//sub x/, /spl theta//sub y/ and /spl theta//sub z/ by 99.1%, 99.1% and 92.8% respectively.

Hysteresis of a piezoelectric actuator is rate dependent. Most hysteresis models are based on elementary rate independent operators and are not suitable for modeling actuator behavior across a wide frequency band. This work proposes a rate dependent modified Prandtl-Ishlinskii (PI) operator to account for the hysteresis of a piezoelectric actuator at varying frequency. We have shown experimentally that the relationship between the slope of the hysteretic loading curve and the rate of control input can be modeled by a linear function. The proposed rate-dependent hysteresis model is implemented for open-loop control of a piezoelectric actuator. In experiments tracking multi-frequency nonstationary motion profiles, it consistently outperforms its rate-independent counterpart by a factor of two in maximum error and a factor of three in rms error.

This paper describes research in active instruments for enhanced accuracy in microsurgery. The aim is to make accuracy enhancement as transparent to the surgeon as possible. Rather than using a robotic arm, we have taken the novel approach of developing a handheld instrument that senses its own movement, distinguishes between desired and undesired motion, and deflects its tip to perform active compensation of the undesired component. The research has therefore required work in quantification and modeling of instrument motion, filtering algorithms for tremor and other erroneous movements, and development of handheld electromechanical systems to perform active error compensation. The paper introduces the systems developed in this research and presents preliminary results.

Presents the development and. initial experimental results of the first prototype of Micron, an active hand-held instrument to sense and compensate physiological tremor and other unwanted movement during vitreoretinal microsurgery. The instrument incorporates six inertial sensors, allowing the motion of the tip to be computed. The motion captured is processed to discriminate between desired and undesired components of motion. Tremor canceling is implemented via the weighted-frequency Fourier linear combiner (WFLC) algorithm, and compensation of non-tremorous error via a neural network technique is being investigated. The instrument tip is attached to a three-degree-of-freedom parallel manipulator with piezoelectric actuation. The actuators move the tool tip in opposition to the tremor, thereby suppressing the erroneous motion. Motion canceling experiments with oscillatory motions in the frequency band of physiological tremor show that Micron is able to reduce error amplitude by 45.3% in 1-D tests and 37.2% in 3-D tests.

We present a novel application of a neural network to augment manual precision by cancelling involuntary motion. This method may be applied in microsurgery, using either a telerobotic approach or active compensation in a handheld instrument. A feedforward neural network is trained to input the measured trajectory of a handheld tool tip and output the intended trajectory. Use of the neural network decreases rms error in recordings from four subjects by an average of 43.9%.