Knowledge of the tip contact force in continuum robots, which are often used as medical instruments, is critical for clinical applications. It enhances the interventionalist's decision-making, navigation efficiency, and procedural safety. However, accurately determining the tip contact force in conventionally sized instruments remains challenging. This study introduces a learning-based method for estimating the external contact force at the tip of a continuum robot. By leveraging curvature and bending angle data from a multi-core fiber equipped with fiber Bragg gratings (FBGs) embedded inside the Nitinol tube, the method maps these inputs to the corresponding tip force in 3D. Experiments conducted on an FBG-embedded Nitinol rod validate the feasibility of the proposed method, yielding Mean Squared Error (MSE), Mean Absolute Error (MAE), and Root Mean Squared Error (RMSE) values of 20.9$\left(m N^{2}\right), 2.7(m N)$, and$4.6(m N)$, respectively, which represent a 26 % improvement compared to the learning-based vision methodology.

ABSTRACT To achieve efficient querying of BIM products in large‐scale virtual scenes, this study introduces a Weisfeiler‐Lehman (WL) kernel augmented representation for Building Information Modeling(BIM) products based on Product Attributed Graphs (PAGs). Unlike conventional data‐driven approaches that demand extensive labeling and preprocessing, our method directly processes raw BIM product data to extract stable semantic and geometric features. Initially, a PAG is constructed to encapsulate product features. Subsequently, a WL kernel enhanced multi‐channel node aggregation strategy is employed to integrate BIM product attributes effectively. Leveraging the bijective relationship in graph isomorphism, an unsupervised convergence mechanism based on attribute value differences is established. Experiments demonstrate that our method achieves convergence within an average of 3 iterations, completes graph isomorphism testing in minimal time, and attains an average query accuracy of 95%. This approach outperforms 1‐WL and 3‐WL methods, especially in handling products with topologically isomorphic but oppositely attributed spaces.

Dexterity is demanded for an endoscopic tool to handle complicated procedures in neurosurgery, e.g., removing diseased tissue from inside the deep brain along a tortuous path. Current robotic tools are either rigid or lack wristed motion ability at the tip, leading to limited usage in minimally invasive procedures. In this paper, a hybrid steerable robot with a magnetic wristed forceps is proposed to provide enhanced dexterity for endoscopic epilepsy surgery. A set of three precurved Nitinol tubes with concentric deployment, called a concentric tube robot (CTR), serves as a 6 degrees-of-freedom (DoF) robotic positioner. The magnetic wristed forceps is composed of a rotational wrist joint, and forceps at the tip, both of which are actuated remotely by magnetic fields. The magnetic wrist and forceps provide an extra rotational DoF and a gripping DoF on top of the CTR, respectively. The magnetic wrist and gripper are designed to have a hollow channel along their common axis, inside which a soft tube is deployed as a second functional tool for irrigation or suction. An electromagnetic navigation system (eMNS) with 8 coils is used to create the quasi-static magnetic fields. Experimental characterization of the robot kinematics is performed and the results show the mean motion error of CTR is 2.8 mm. The workspace is also analyzed and results indicate that the proposed hybrid robot has a significantly larger reachable area compared to the one of the CTR alone. Mock epilepsy procedures are performed on a brain phantom to validate the feasibility of the hybrid robot for neurosurgery applications.

Abstract The increased growth of building complexity confronts the limited device resources, especially for lightweight personal digital devices. The research of handling these complex buildings with limited resources has become a pivotal research topic. This paper explores the task of lightweight indoor path planning in complex building information modeling (BIM) buildings. Both the environment modeling and path searching are important addressed through indoor path planning with a lightweight web‐based approach. A lightweight framework is designed in which the complex indoor path planning of a whole BIM building can be simplified into many local path searching of the building's interior spaces and all the online computation is decreased within no more than two simpler ones. In addition, an automatically indoor environment modeling method is proposed to partition the BIM building into many local interior spaces, constructing their spatial relationship and local grid map in a multilayered network. In the end, a heuristics optimized A star algorithm is implemented to speed up the shortest local path searching for the case of the path ends in a narrow corner. The final experiment shows that the online computational cost can be reduced greatly and dynamically to an average of 2 s when searching for the shortest path in a complex BIM building.

Retinal vein cannulation is a promising approach for treating retinal vein occlusion that involves injecting medicine into the occluded vessel to dissolve the clot. The approach remains largely unexploited clinically due to surgeon limitations in detecting interaction forces between surgical tools and retinal tissue. In this paper, a dual force constraint controller for robot-assisted retinal surgery was presented to keep the tool-to-vessel forces and tool-to-sclera forces below prescribed thresholds. A cannulation tool and forceps with dual force-sensing capability were developed and used to measure force information fed into the robot controller, which was implemented on existing Steady Hand Eye Robot platforms. The robotic system facilitates retinal vein cannulation by allowing a user to grasp the target vessel with the forceps and then enter the vessel with the cannula. The system was evaluated on an eye phantom. The results showed that, while the eyeball was subjected to rotational disturbances, the proposed controller actuates the robotic manipulators to maintain the average tool-to-vessel force at 10.9 mN and 13.1 mN and the average tool-to-sclera force at 38.1 mN and 41.2 mN for the cannula and the forcpes, respectively. Such small tool-to-tissue forces are acceptable to avoid retinal tissue injury. Additionally, two clinicians participated in a preliminary user study of the bimanual cannulation demonstrating that the operation time and tool-to-tissue forces are significantly decreased when using the bimanual robotic system as compared to freehand performance.

A fundamental challenge in retinal surgery is safely navigating a surgical tool to a desired goal position on the retinal surface while avoiding damage to surrounding tissues, a procedure that typically requires tens-of-microns accuracy. In practice, the surgeon relies on depth-estimation skills to localize the tool-tip with respect to the retina in order to perform the tool-navigation task, which can be prone to human error. To alleviate such uncertainty, prior work has introduced ways to assist the surgeon by estimating the tooltip distance to the retina and providing haptic or auditory feedback. However, automating the tool-navigation task itself remains unsolved and largely unexplored. Such a capability, if reliably automated, could serve as a building block to streamline complex procedures and reduce the chance for tissue damage. Towards this end, we propose to automate the tool-navigation task by learning to mimic expert demonstrations of the task. Specifically, a deep network is trained to imitate expert trajectories toward various locations on the retina based on recorded visual servoing to a given goal specified by the user. The proposed autonomous navigation system is evaluated in simulation and in physical experiments using a silicone eye phantom. We show that the network can reliably navigate a needle surgical tool to various desired locations within 137 μm accuracy in physical experiments and 94 μm in simulation on average, and generalizes well to unseen situations such as in the presence of auxiliary surgical tools, variable eye backgrounds, and brightness conditions.

One of the significant challenges of moving from manual to robot-assisted retinal surgery is the loss of perception of forces applied to the sclera (sclera forces) by the surgical tools. This damping of force feedback is primarily due to the stiffness and inertia of the robot. The diminished perception of tool-to-eye interactions might put the eye tissue at high risk of injury due to excessive sclera forces or extreme insertion of the tool into the eye. In the present study therefore a 1-dimensional adaptive control method is customized for 3-dimensional control of sclera force components and tool insertion depth and then implemented on the velocity-controlled Johns Hopkins Steady-Hand Eye Robot. The control method enables the robot to perform autonomous motions to make the sclera force and/or insertion depth of the tool tip to follow pre-defined desired and safe trajectories when they exceed safe bounds. A robotic light pipe holding application in retinal surgery is also investigated using the adaptive control method. The implementation results indicate that the adaptive control is able to achieve the imposed safety margins and prevent sclera forces and insertion depth from exceeding safe boundaries.

Retinal microsurgery is technically demanding and requires high surgical skill with very little room for manipulation error. During surgery the tool needs to be inserted into the eyeball while maintaining constant contact with the sclera. Any unexpected manipulation could cause extreme tool-sclera contact force (scleral force) thus damage the sclera. The introduction of robotic assistance could enhance and expand the surgeon's manipulation capabilities during surgery. However, the potential intra-operative danger from surgeon's misoperations remains difficult to detect and prevent by existing robotic systems. Therefore, we propose a method to predict imminent unsafe manipulation in robot-assisted retinal surgery and generate feedback to the surgeon via auditory substitution. The surgeon could then react to the possible unsafe events in advance. This work specifically focuses on minimizing sclera damage using a force-sensing tool calibrated to measure small scleral forces. A recurrent neural network is designed and trained to predict the force safety status up to 500 milliseconds in the future. The system is implemented using an existing "steady hand" eye robot. A vessel following manipulation task is designed and performed on a dry eye phantom to emulate the retinal surgery and to analyze the proposed method. Finally, preliminary validation experiments are performed by five users, the results of which indicate that the proposed early warning system could help to reduce the number of unsafe manipulation events.

Retinal surgery involves manipulating very delicate tissues within the confined area of eyeball. In such demanding practices, patient involuntary head movement might abruptly raise tool-to-eyeball interaction forces which would be detrimental to eye. This study is aimed at implementing different force control strategies and evaluating how they contribute to attaining sclera force safety while patient head drift is present. To simulate patient head movement, a piezoelectric-actuated linear stage is used to produce random motions in a single direction in random time intervals. Having an eye phantom attached to the linear stage then an experienced eye surgeon is asked to manipulate the eye and repeat a mock surgical task both with and without the assist of the Steady-Hand Eye Robot. For the freehand case, warning sounds were provided to the surgeon as auditory feedback to alert him about excessive slclra forces. For the robot-assisted experiments two variants of an adaptive sclera force control and a virtual fixture method were deployed to see how they can maintain eye safety under head drift circumstances. The results indicate that the developed robot control strategies are able to compensate for head drift and keep the sclera forces under safe levels as well as the free hand operation.

Retinal microsurgery is technically demanding and requires high surgical skill with very little room for manipulation error. The introduction of robotic assistance has the potential to enhance and expand a surgeon's manipulation capabilities during retinal surgery, i.e., improve precision, cancel physiological hand tremor, and provide sensing information. However, surgeon performance may also be negatively impacted by robotic assistance due to robot structural stiffness and nonintuitive controls. In complying with robotic constraints, the surgeon loses the dexterity of the human hand. In this paper, we present a preliminary experimental study to evaluate user behavior when affected by robotic assistance during mock retinal surgery. In these experiments user behavior is characterized by measuring the forces applied by the user to the sclera, the tool insertion/retraction speed, the tool insertion depth relative to the scleral entry point, and the duration of surgery. The users' behavior data is collected during three mock retinal surgery tasks with four users. Each task is conducted using both freehand and robot-assisted techniques. The univariate user behavior and the correlations of multiple parameters of user behavior are analyzed. The results show that robot assistance prolongs the duration of the surgery and increases the manipulation forces applied to sclera, but refines the insertion velocity and eliminates hand tremor.