In modern healthcare, the demand for autonomous robotic assistants has grown significantly, particularly in the operating room, where surgical tasks require precision and reliability. Robotic scrub nurses have emerged as a promising solution to improve efficiency and reduce human error during surgery. However, challenges remain in terms of accurately grasping and handing over surgical instruments, especially when dealing with complex objects in dynamic environments. In this work, we introduce RoboNurse-VLA, a novel robotic scrub nurse system based on a Vision-Language-Action (VLA) model. RoboNurse-VLA integrates Segment Anything Model 2 (SAM 2) and Llama 2, leveraging an LLM head to enhance reasoning capabilities. By combining SAM 2’s mask generation with Llama 2’s advanced reasoning, RoboNurse-VLA can accurately interpret task requirements, identify optimal grasping points, and determine appropriate handover poses. Designed for real-time operation, RoboNurse-VLA enables precise grasping and seamless handover of surgical instruments based on voice commands from the surgeon. Utilizing state-of-the-art vision and language models, it effectively addresses challenges related to object detection, pose optimization, and handling difficult-to-grasp instruments. Extensive evaluations demonstrate that RoboNurse-VLA outperforms existing models, achieving high success rates in surgical instrument handovers, even for previously unseen tools and complex objects. This work represents a significant advancement in autonomous surgical assistance, highlighting the potential of VLA models for real-world medical applications. More details can be found at https:// robonurse-vla.github.io.

This paper is concerned with the issue of the kinematic model-based bending control for the magnetically actuated robotic endoscope and its application for automatic retroflexion. By the utilization of the Cosserat rod theory and the transformation in the magnetic tip of the endoscope, the comprehensive kinematic model of the magnetically-actuated robotic endoscope is established. Afterward, a magnetic control scheme for the bending motion is proposed by co-developing an error feedback PID control strategy and the model-based feedback approach. Moreover, as one unique kind of bending motion, retroflexion is taken into account, and the strategy aimed at the bid of compact space retroflexion is presented by virtue of the introduction of serial waypoints pursuing the position of the magnetic tip being close to the midline as possible. Eventually, the developed modeling and bending control scheme and the compact space retroflexion strategy are examined in a magnetically actuated robotic endoscope system to manifest the effectiveness and applicability of the theoretical approach. The experimental results indicate that the designed controller can drive the endoscope to bend to the desired pose and show a reduction of about 47.01% in the sweeping area and 79.25% in the last distance to midline achieved by conducting compact space retroflexion in comparison to “U” type one.

In this paper, we proposed an autonomous robotic flexible endoscope system for the laparoscopic bariatric surgery (LBS). This system comprises a UR5 robot and a flexible endoscope equipped with a novel continuum joint, named reinforced double helix continuum mechanism. Compared with the simple helix structure, the compressional and torsional stiffness of the proposed joint are improved significantly. To automate the robotic flexible endoscope, image-based visual servoing technique is employed. A deep learning algorithm named TernausNet-16 is improved and incorporated into the control framework to detect surgical instruments inside the camera view. The experimental studies verified the effectiveness and feasibility of the robotic flexible endoscope system for the visual serviong control scheme assisted by deep learning methods.