We introduce a novel hybrid vision/force control strategy for robotic devices designed to obtain clear and consistent images using probe-based confocal laser endomicroscopy (pCLE). Due to the variable nature of tissue characteristics encountered during pCLE imaging, the conventional approach of pre-setting forces or focus metrics for either force or vision control is often impractical and inadequate. To address this, our strategy employs a blur metric called the CR score to assess the level of blur in pCLE images, enabling the attainment of clear and focused images. At the onset of a pCLE scan, the system autonomously determines the target CR score for vision control, in tandem with a real-time peak detection algorithm. Concurrently, force control is applied judiciously to prevent excessive force on the tissue, adjusting to ensure minimal force is applied, thus preserving image focus. This innovative approach facilitates seamless transitions between vision and force control, depending on the imaging conditions, thereby ensuring the acquisition of consistent pCLE images with minimal force. Our method marks a notable improvement over conventional PID force control techniques. By dynamically adjusting target forces and minimizing force application during operation, we not only enhance the precision and quality of pCLE imaging but also eliminate the dependency on manual pre-settings.

In this study, we present a visual servo control framework for fully automated nasopharyngeal swab robots. The proposed framework incorporates a deep learning-based nostril detection with a cascade approach to reliably identify the nostrils with high accuracy in real time. In addition, a partitioned visual servoing scheme that combines image-based visual servoing with axial control is formulated for accurately positioning the sampling swabs at the nostril with a multi-DOF robot arm. As the visual servoing is designed to minimize an error between the detected nostril and the swab, it can compensate for potential errors in real operation, such as positioning error by inaccurate camera-robot calibration and kinematic error by unavoidable swab deflection. The performance of the visual servo control was tested on a head phantom model for 30 unused swabs, and then compared with a method referring to only the 3D nostril target for control. Consequently, the swabs reached the nostril target with less than an average error of 1.2±0.5 mm and a maximum error of 2.0 mm via the visual servo control, while the operation without visual feedback yielded an average error of 10.6±2.3 mm and a maximum error of 16.2 mm. The partitioned visual servoing allows the swab to rapidly converge to the nostril target within 1.0 s without control instability. Finally, the swab placement at the nostril among the entire procedure of fully automated NP swab was successfully demonstrated on a human subject via the visual servo control.

This paper presents the design and control of a fully handheld robot for robot-assisted microsurgery. The handheld robot incorporates a miniature 6-PUS parallel micromanipulator that can impose a remote center of motion (RCM) at the incision point of entry during microsurgery. An optimization framework is formulated to determine the geometric parameters of the micromanipulator. The optimization aims to minimize force applied on actuation modules by lateral load at the RCM. The optimization yields a base diameter of 16 mm, a top diameter of 12.6 mm, and a connecting link of 9.4 mm, which offers a cylindrical workspace 4-mm wide and 3-mm high with a 5-mm travel in linear actuation. An order of magnitude higher force capability is attained in the smaller form factor compared to the latest handheld micromanipulator. We built the fully handheld version of the microsurgical robot by incorporating embedded electronics and an EM tracker for sensing the 6-DOF pose of hand motion. The real-time control framework of the handheld robot is also presented, including motion filter for active tremor cancellation. As a result, the handheld robot can tolerate side loads up to 5.0 N for a lateral load applied at the RCM without significant degradation in control. Finally, the robot-aided operation with the active tremor cancellation shows a significant peak-force reduction compared to unaided operation during the task of maintaining contact force.

We develop a miniature suction-gripper with the goal to realize the novel robotic surgical instrument that can grip slippery and flexible peripheral nerves. In developing the instrument, we place a priority on devising the method that can robustly grip the nerve bundles during the surgical operation for the peripheral nerve. Also, we concentrate to investigate the working principle being able to minimize nerve damages that might be caused when manipulating the nerve. In this study, as the most suitable method to achieve the goal, we scheme to utilize the suction mechanism. Because it can non-invasively grip the nerve based on negative pressure, no external force is applied to the nervous tissues. Therefore the peripheral nerve can be manipulated without serious nerve damage (e.g. crush injury and stretch injury). To improve the gripping ability of the proposed suction gripper, two different types of microneedle arrays are applied to the suction-tips: passive-microneedle (PMN) arrays and active-microneedle (AMN) arrays. Since the most outer membrane of the nerve can be anchored by the penetrated PMN and AMN, the gripper can grip the nerve more robustly. The designed suction-gripper is fabricated as a functional prototype, and its working performances are assessed with in-vitro and in-vivo animal experiments. The experimental results well demonstrate the practical effectiveness of the proposed method and its applicability to the neurosurgical robot for the peripheral nerve.

Motion capture systems are gaining much attention in various fields, including entertainment, medical and sports fields. Although many types of motion capture sensor have been emerging, they have limitations and disadvantages such as occlusion, drift and interference by electromagnetic fields. Here, we introduce the novel wearable motion capture system using fiber Bragg gratings (FBGs) sensors. Since the human joints have different degrees of freedom (DOF), we developed three types of sensors to reconstruct the human body motion from the strains induced on the FBGs. First, a shape sensor using three fibers provides the position and orientation of joints in three dimensional space. Second, we introduce the angle sensor which is capable of measuring bending angle with high curvature using single fiber. Lastly, to detect the twisting of joints, a sensor with fiber attached on a soft material spirally is used. With the optical fiber based motion capture sensors, we reconstruct the motion of arm in realtime. In detail, the joints of the arm include the sternoclavicular, acromioclavicular, shoulder and elbow. By arranging the three types of sensors on the joints in accordance with the DOF, the accuracy of the reconstructed motion is evaluated, resulting in an average error below 2.42°. Finally, to prove the feasibility of applying in virtual reality, we successfully manipulate the virtual avatar in real-time.

In this paper we describe work towards retinal vessel cannulation using an actively stabilized handheld robot, guided by monocular vision. We employ a previously developed monocular camera based surface reconstruction method using automated laser beam scanning over the retina. We use the reconstructed plane to find a coordinate transform between the 2D image plane coordinate system and the global 3D frame. Within a hemispherical region around the target, we use motion scaling for higher precision. The contribution of this work is the homography matrix estimation using monocular vision and application of the previously developed laser surface reconstruction to Micron guided vein cannulation. Experiments are conducted in a wet eye phantom to show the higher accuracy of the surface reconstruction as compared to standard stereo reconstruction. Further, experiments to show the increased surgical accuracy due to motion scaling are also carried out.

This paper presents robot-aided intraocular laser surgery using a handheld robot known as Micron. The micromanipulator incorporated in Micron enables visual servoing of a laser probe, while maintaining a constant distance of the tool tip from the retinal surface. The comparative study was conducted with various control methods for evaluation of robot-aided intraocular laser surgery.

This paper presents a technique for automated intraocular laser surgery using a handheld micromanipulator known as Micron. The novel handheld manipulator enables the automated scanning of a laser probe within a cylinder of 4 mm long and 4 mm in diameter. For the automation, the surface of the retina is reconstructed using a stereomicroscope, and then preplanned targets are placed on the surface. The laser probe is precisely located on the target via visual servoing of the aiming beam, while maintaining a specific distance above the surface. In addition, the system is capable of tracking the surface of the eye in order to compensate for any eye movement introduced during the operation. We compared the performance of the automated scanning using various control thresholds, in order to find the most effective threshold in terms of accuracy and speed. Given the selected threshold, we conducted the handheld operation above a fixed target surface. The average error and execution time are reduced by 63.6% and 28.5%, respectively, compared to the unaided trials. Finally, the automated laser photocoagulation was demonstrated also in an eye phantom, including compensation for the eye movement.

This paper presents the characterization and comparison of physiological tremor for pointing tasks in multiple environments, as a baseline for performance evaluation of microsurgical robotics. Previous studies have examined the characteristics of physiological tremor under laboratory settings as well as different operating conditions. However, different test methods make the comparison of results across trials and conditions difficult. Two vitroretinal microsurgeons were evaluated while performing a pointing task with no entry-point constraint, constrained by an artificial eye model, and constrained by a rabbit eye in vivo. For the three respective conditions the 3D RMS positioning error was 144 μm, 258 μm, and 285 μm, and maximum 3D error was 349 μm, 647 μm, and 696 μm. A spectral analysis was also performed, confirming a distinct peak near in the 6-12 Hz frequency range, characteristic of hand tremor during tasks in all three environments.

This paper presents the design and analysis of a handheld manipulator for vitreoretinal microsurgery and other biomedical applications. The design involves a parallel micromanipulator utilizing six piezoelectric linear actuators, combining compactness with a large range of motion and relatively high stiffness. Given the available force of the actuators, the overall dimension of the micromanipulator was optimized considering realistic external loads on a remote center of motion representing the point of expected contact with the sclera of the eye during microsurgery. Based on optimization and workspace analysis, a benchtop version of the micromanipulator was built with a base diameter of 25 mm and a height of 50 mm. It provides a hemispherical workspace of 4.0 mm diameter at the tool tip. The manipulation performance of the constructed manipulator was measured under a lateral load applied at the remote center of motion. The micromanipulator tolerated side loads up to 200 mN.

A precise and implantable micromanipulator is presented for automatically advancing electrodes during single unit recordings in freely-behaving animals. The modular design and enhanced clamping mechanism with simple mechanical components are designed to provide reliable linear motion using a piezo motor with a stroke of 3 mm. To be specific, a closed loop control system, based on the position feedback from a magnetoresistive (MR) sensor, was implemented to overcome the non-linear characteristics of the piezo motor and to locate electrodes precisely at the targeted position with the accuracy of 1 μm, even under load. The weight of the micromanipulator is only 0.84 g when it is fully assembled with the MR sensor, PCBs, and connectors. In addition, a protective cover is employed to prevent breakage during semi-chronic recording. The positioning performance of the micromanipulator was tested at various loading conditions using various control methods. Finally, the activities of a single unit were isolated successfully using small step adjustments, such as 1 to 5 μm, in freely-moving mice.