Precise needle alignment is essential for percutaneous needle insertion in robotic ultrasound-guided procedures. However, inherent challenges such as speckle noise, needle-like artifacts, and low image resolution complicate robust needle detection, which is essential for alignment in ultrasound images. These issues become particularly problematic when visibility is reduced or lost, diminishing the effectiveness of visual-based needle alignment methods. In this paper, we propose a method to restore effectively when the ultrasound imaging plane and the needle insertion plane are misaligned. Unlike many existing approaches that rely heavily on needle visibility in ultrasound images, our method uses a more robust feature by periodically vibrating the needle using a mechanical system. Specifically, we propose a new vibration-based energy metric that remains effective even when the needle is fully out of plane. Using this metric, we develop an elegant control strategy to reposition the ultrasound probe in response to misalignments between the imaging plane and the needle insertion plane in both translation and rotation. Experiments conducted on ex-vivo porcine tissue samples using a dual-arm robotic ultrasound-guided needle insertion system demonstrate the effectiveness of the proposed approach. The experimental results show the translational error of 0.41±0.27 mm and the rotational error of 0.51±0.19 degrees.

As one of the effective closed-loop control methods, visual servoing control methods are widely applied to continuum robots. However, existing visual servoing control methods mostly focus on accurate control of the robot’s end-effector, with less consideration given to the robot’s shape. In this work, a spatial position-based visual servoing obstacle-avoidable shape control framework for an 11-degree-of-freedom (DOF) hybrid continuum robot is proposed. In the control framework, a set of markers representing the shape of the continuum robot are measured and two spatial arcs are used to fit the shape. When controlling the redundant DOFs of the robot, position-based visual servoing shape control combined with obstacle avoidance is formulated as a quadratic programming problem, yielding the optimal solution at each sample time for the joint velocity vector of the 11-DOF hybrid continuum robot. Several experiments are conducted to validate the proposed control framework, which indicates the accuracy of the shape control achieves 0.88 mm.

This paper proposes a decentralized adaptive fuzzy event-triggered fault-tolerant control scheme for nonlinear switched interconnected systems under arbitrary switchings. A novel state observer is crafted to estimate unmeasured states by using the faulty output signal. Interval type-2 fuzzy logic systems are adopted to deal with uncertain nonlinearities. The Nussbaum gain technique and a novel fault compensation mechanism are utilized to deal with actuator and sensor faults. A switching threshold event-triggered control strategy is developed to guarantee that all closed-loop signals are bounded, meanwhile the Zeno behavior is excluded. Eventually, a practical simulation illustrates the validation of the theoretical findings.

Precise percutaneous needle detection is crucial for ultrasound (US)-guided interventions. However, inherent limitations such as speckles, needle-like artifacts, and low resolution make it challenging to robustly detect needles, especially when their visibility is reduced or imperceptible. To address this challenge, we propose VibNet, a learning-based framework designed to enhance the robustness and accuracy of needle detection in US images by leveraging periodic vibration applied externally to the needle shafts. VibNet integrates neural Short-Time Fourier Transform and Hough Transform modules to achieve successive sub-goals, including motion feature extraction in the spatiotemporal space, frequency feature aggregation, and needle detection in the Hough space. Due to the periodic subtle vibration, the features are more robust in the frequency domain than in the image intensity domain, making VibNet more effective than traditional intensity-based methods. To demonstrate the effectiveness of VibNet, we conducted experiments on distinct ex vivo porcine and bovine tissue samples. The results obtained on porcine samples demonstrate that VibNet effectively detects needles even when their visibility is severely reduced, with a tip error of ${1}.{61}\pm {1}.{56}~\textit {mm}$ compared to ${8}.{15}\pm {9}.{98}~\textit {mm}$ for UNet and ${6}.{63}\pm {7}.{58}~\textit {mm}$ for WNet, and a needle direction error of ${1}.{64}\pm {1}.{86}^{\circ }$ compared to ${9}.{29}~\pm ~{15}.{30}^{\circ }$ for UNet and ${8}.{54}~\pm ~{17}.{92}^{\circ }$ for WNet. Code: https://github.com/marslicy/VibNet.

This paper proposes an interactive navigation framework by using large language and vision-language models, allowing robots to navigate in environments with traversable obstacles. We utilize the large language model (GPT-3.5) and the open-set Vision-language Model (Grounding DINO) to create an action-aware costmap to perform effective path planning without fine-tuning. With the large models, we can achieve an end-to-end system from textual instructions like "Can you pass through the curtains to deliver medicines to me?", to bounding boxes (e.g., curtains) with action-aware attributes. They can be used to segment LiDAR point clouds into two parts: traversable and untraversable parts, and then an action-aware costmap is constructed for generating a feasible path. The pre-trained large models have great generalization ability and do not require additional annotated data for training, allowing fast deployment in the interactive navigation tasks. We choose to use multiple traversable objects such as curtains and grasses for verification by instructing the robot to traverse them. Besides, traversing curtains in a medical scenario was tested. All experimental results demonstrated the proposed framework’s effectiveness and adaptability to diverse environments.

Flexible endoscopes are commonly adopted in narrow and confined anatomical cavities due to their higher reachability and dexterity. However, prolonged and unintuitive manipulation of these endoscopes leads to an increased workload on surgeons and risks of collision. To address these challenges, this paper proposes a CT-guided control framework for the diagnosis of maxillary sinusitis by using a robotic flexible endoscope. In the CT-guided control framework, a feasible path to the target position in the maxillary sinus cavity for the robotic flexible endoscope is designed. Besides, an optimal control scheme is proposed to autonomously control the robotic flexible endoscope to follow the feasible path. This greatly improves the efficiency and reduces the workload for surgeons. Several experiments were conducted based on a widely utilized sinus phantom, and the results showed that the robotic flexible endoscope can accurately and autonomously follow the feasible path and reach the target position in the maxillary sinus cavity. The results also verified the feasibility of the CT-guided control framework, which contributes an effective approach to early diagnosis of sinusitis in the future.

Falling cat problem is well-known where cats show their super aerial reorientation capability and can land safely. For their robotic counterparts, a similar falling quadruped robot problem, has not been fully addressed, although achieving safe landing as the cats has been increasingly investigated. Unlike imposing the burden on landing control, we approach to safe landing of falling quadruped robots by effective flight phase control. Different from existing work like swinging legs and attaching reaction wheels or simple tails, we propose to deploy a 3-DoF morphable inertial tail on a medium-size quadruped robot. In the flight phase, the tail with its maximum length can self-right the body orientation in 3D effectively; before touch-down, the tail length can be retracted to about 1/4 of its maximum for impressing the tail's side-effect on landing. To enable aerial reorientation for safe landing in the quadruped robots, we design a control architecture, which is verified in a high-fidelity physics simulation environment with different initial conditions. Experimental results on a customized flight-phase test platform with comparable inertial properties are provided and show the tail's effectiveness on 3D body reorientation and its fast retractability before touch-down. An initial falling quadruped robot experiment is shown, where the robot Unitree A1 with the 3-DoF tail can land safely subject to non-negligible initial body angles.

Projected Inverse Dynamics Control (PIDC) is commonly used in robots subject to contact, especially in quadrupedal systems. Many methods based on such dynamics have been developed for quadrupedal locomotion tasks, and only a few works studied simple interactions between the robot and environment, such as pressing an E-stop button. To facilitate the interaction requiring exact force control for safety, we propose a novel interaction force control scheme for under-actuated quadrupedal systems relying on projection techniques and Quadratic Programming (QP). This algorithm allows the robot to apply a desired interaction force to the environment without using force sensors while satisfying physical constraints and inducing minimal base motion. Unlike previous projection-based methods, the QP design uses two selection matrices in its hierarchical structure, facilitating the decoupling between force and motion control. The proposed algorithm is verified with a quadrupedal robot in a high-fidelity simulator. Compared to the QP designs without the strategy of using two selection matrices and the PIDC method for contact force control, our method provided more accurate contact force tracking performance with minimal base movement, paving the way to approach the exact interaction force control for underactuated quadrupedal systems.

Needle picking is a challenging manipulation task in robot-assisted surgery due to the characteristics of small slender shapes of needles, needles' variations in shapes and sizes, and demands for millimeter-level control. Prior works, heavily relying on the prior of needles (e.g., geometric models), are hard to scale to unseen needles' variations. In this paper, we present the first end- to-end learning method to train deep visuomotor policy for needle picking. Concretely, we propose DreamerfD to maximally leverage demonstrations to improve the learning efficiency of a state-of-the-art model-based reinforcement learning method, DreamerV2; Since Variational Auto-Encoder (VAE) in DreamerV2 is difficult to scale to high-resolution images, we propose Dynamic Spotlight Adaptation to represent control-related visual signals in a low-resolution image space; Virtual Clutch is also proposed to reduce per-formance degradation due to significant error between prior and posterior encoded states at the beginning of a rollout. We conducted extensive experiments in simulation to evaluate the performance, robustness, in-domain variation adaptation, and effectiveness of individual components of our method. Our method, trained by 8k demonstration timesteps and 140k online policy timesteps, can achieve a remarkable success rate of 80%. Furthermore, our method effectively demonstrated its superiority in generalization to unseen in-domain variations including needle variations and image disturbance, highlighting its robustness and versatility. Codes and videos are available at https://sites.google.com/view/DreamerfD.

The rigid and straight nature of conventional surgical drills and screwdrivers makes it difficult to access the posterior mandible for fracture reduction without the creation of facial incisions. To assist transoral mandibular fracture reduction in hard-to-reach areas, we propose a shared-control dexterous robotic system. The end effector of this system is an articulated drilling/screwing tool to provide distal dexterity. This system uses an admittance-control-based approach to provide precision and stability during shared-control hole-drilling processes. A cadaver study showed the efficacy of the proposed system to assist plate fixation in the reduction of mandibular fractures. The proposed articulated surgical tool was capable of drilling holes in and driving screws into the mandible of a cadaver head. In addition, the shared-control robotic system ensured that the drill moved along its axial direction, leading to stable and precise hole drilling.

In robotic deformable object manipulation (DOM) applications, constraints arise commonly from environments and task-specific requirements. Enabling DOM with constraints is, therefore, crucial for its deployment in practice. However, dealing with constraints turns out to be challenging due to many inherent factors, such as inaccessible deformation models of deformable objects (DOs) and varying environmental setups. This article presents a systematic manipulation framework for DOM subject to constraints by proposing a novel path set planning and tracking scheme. First, constrained DOM tasks are formulated into a versatile optimization formalism, which enables dynamic constraint imposition. Because of the lack of the local optimization objective and high state dimensionality, the formulated problem is not analytically solvable. To address this, planning of the path set, which collects paths of DO feedback points, is proposed subsequently to offer feasible path and motion references for the DO in constrained setups. Both theoretical analyses and computationally efficient algorithmic implementation of path set planning are discussed. Lastly, a control architecture combining path set tracking and constraint handling is designed for task execution. The effectiveness of our methods is validated in a variety of DOM tasks with constrained experimental settings.

Image processing has significantly extended the practical value of the eye-in-hand camera, enabling and promoting its applications for quantitative measurement. However, fully vision-based pose estimation methods sometimes encounter difficulties in handling cases with deficient features. In this article, we fuse visual information with the sparse strain data collected from a single-core fiber inscribed with fiber Bragg gratings (FBGs) to facilitate continuum robot pose estimation. An improved extreme learning machine algorithm with selective training data updates is implemented to establish and refine the FBG-empowered (F-emp) pose estimatoronline. The integration of F-emp pose estimation can improve sensing robustness by reducing the number of times that visual tracking is lost given moving visual obstacles and varying lighting. In particular, this integration solves pose estimation failures under full occlusion of the tracked features or complete darkness. Utilizing the fused pose feedback, a hybrid controller incorporating kinematics and data-driven algorithms is proposed to accomplish fast convergence with high accuracy. The online-learning error compensator can improve the target tracking performance with a 52.3%–90.1% error reduction compared with constant-curvature model-based control, without requiring fine model-parameter tuning and prior data acquisition.

Soft robots have great potential in surgical applications due to their compliance and adaptability to their environment. However, their flexibility and nonlinearity bring challenges for precise modeling, sensing, and control, especially in constrained cavities. In this article, a simple, compact two-segment soft robot for flexible laser ablation is proposed. The proximal hydraulic-driven segment can offer omnidirectional bending so as to navigate toward lesions. The distal segment driven by tendons enables precise, fast steering of laser collimator for laser sweeping on lesion targets. The dynamics of such mechanical steering motion can be enhanced with a metal spring backbone integrated along the collimator, thus facilitating the control with certain linearity and responsiveness. A soft robot modeling and control scheme based on Koopman operators is proposed. We also design a disturbance observer so as to incorporate the controller feedback with real-time fiber optic shape sensing. Experimental validation is conducted on simulated orex-vivolaser ablation tasks, thus evaluating our control strategies in laser path following across various contours/patterns. As a result, such a simple compact laser manipulation can perform up to 6 Hz sweeping with precision of path following errors below 1 mm. Such modeling and control scheme could also be used on an endoscopic laser ablation robot with unsymmetric mechanism driven by two tendons.

Existing robotic endoscopes for laparoscopic surgery, predominantly rigid or limited in dexterity, occupy a large motion space1, The large occupied motion space necessitates large incisions and reduces the motion space for surgeons to simultaneously operate other surgical instruments. Meanwhile, surgeons only have limited view adjustment capability to avoid occlusion and they often have to lift/push some organs to observe occluded target lesions in some operations such as cholecystectomy. The situation gets worse when the operations are on obese patients. In this paper, we develop a novel dexterous robotic flexible endoscope (DRFE), which is comprised of a concentric cable-driven structure and a 2-DoF articulated joint attached to the end of DRFE, for laparoscopic surgery. The proposed design occupies much less motion space both inside and outside human body as compared to conventional robotic flexible endoscopes. When used in surgery, the part of the endoscope outside the body can remain still, which reduces the risk of expanding the incision and simplifies the structure of the remote center mechanism. Simulation and experimental studies are performed to validate the effectiveness of the proposed device in the improvement of vision occlusion and usability. Initial results reveal that the DRFE is highly dexterous and accurate in observing lesions with vision occlusion.

In this paper, we propose an operational space control formulation for a planar N-link underactuated manipulator (PAN–1)1with a passive first joint subject to actuator constraints (N ⩾ 3), covering both stabilization and tracking tasks. Such underactuated manipulators have an inherent first-order nonholonomic constraint, allowing us to project their dynamics to a space consistent with the nonholonomic constraint. Based on the constrained dynamics, we can design operational space controllers with respect to tasks assuming that all joints of the manipulator are active. Due to underactuation, we design a Quadratic Programming (QP) based controller to minimize the error between the desired torque commands and available motor torques in the null space of the constraint, as well as involve the constraint of motor outputs. The proposed control framework was demonstrated by stabilization and tracking tasks in simulations with both planar PA2and PA3manipulators. Furthermore, we verified the controller experimentally using a planar PA2robot.

A single wheel, gyroscopically stabilized robot was developed to provide a dynamic stability for rapid locomotion. It is a sharp-edged wheel actuated by a spinning flywheel for steering and a drive motor for propulsion. The spinning flywheel acts as a gyroscope to stabilize the robot and it can be tilted to achieve steering. In this paper, we present a path following controller for the robot. We first describe the robot motion by a set of configurations using the path curvature. We present a controller for tracking any desired straight line without falling over. For the controller, we first design the linear and steering velocities for driving the robot to the desired straight line through controlling the path curvature. The controller then applies the linear state feedback to stabilize the robot to the predefined lean angle such that the resulting steering velocity of the robot converges to the given steering velocity.

Gyrover is a single wheel, gyroscopically stabilized robot. It is a single wheel connected to a spinning flywheel through a two-link manipulator at the wheel bearing. The nature of the system is nonholonomic, nonlinear and underactuated. In this paper, we first develop a dynamic model and decouple the model with respect to the control inputs. We then study the effect of the flywheel dynamics on stabilizing the single wheel robot via simulation and experiment study. Finally, we design a linear state feedback control law that stabilizes the single wheel robot toward/in different lean angles, so as to control the precession rate. Simulation and experiment study validated the proposed controller as well as the developed dynamic model.

We present a method to model human operator's strategy in controlling a dynamically stabilized robot, Gyrover, which is a single-wheel gyroscopically stabilized robot. We first select the relevant state variables for training from kinematic and dynamic equations. Then, we defined a measure of the sensitivity of each of the state variables with respect to operator's control input by a sensitivity function in order to reduce the number of the state variables required in the model. We experimentally implemented the method and demonstrated that the robot can be automatically controlled using the learned human control model. The work is of significance in abstracting operator's skill for controlling a dynamically stabilized system in generating an automatic control input.

We develop a dynamic model of the steering and actuation mechanism of Gyrover, a single-wheel robot which can be considered as a single wheel, actuated through a spinning flywheel attached through a two-link manipulator at the wheel bearing and a drive motor. The spinning flywheel acts as a gyroscope to stabilize the robot, and at the same time it can achieve steering. We develop a dynamic model, investigate its motion equation, and nonholonomic constraints, and present a simulation study. The work is significant in understanding this type of dynamically stable but statically unstable system, and in developing automatic control of the system.