Accurate pose estimation of surgical tools in Robot-assisted Minimally Invasive Surgery (RMIS) is essential for surgical navigation and robot control. While traditional marker-based methods offer accuracy, they face challenges with occlusions, reflections, and tool-specific designs. Similarly, supervised learning methods require extensive training on annotated datasets, limiting their adaptability to new tools. Despite their success in other domains, zero-shot pose estimation models remain unexplored in RMIS for pose estimation of surgical instruments, creating a gap in generalising to unseen surgical tools. This paper presents a novel 6 Degrees of Freedom (DoF) pose estimation pipeline for surgical instruments, leveraging state-of-the-art zero-shot RGB-D models like the Foundation-Pose and SAM-6D. We advanced these models by incorporating vision-based depth estimation using the RAFT-Stereo method, for robust depth estimation in reflective and textureless environments. Additionally, we enhanced SAM-6D by replacing its instance segmentation module, Segment Anything Model (SAM), with a fine-tuned Mask R-CNN, significantly boosting segmentation accuracy in occluded and complex conditions. Extensive validation reveals that our enhanced SAM-6D surpasses FoundationPose in zero-shot pose estimation of unseen surgical instruments, setting a new benchmark for zero-shot RGB-D pose estimation in RMIS. This work enhances the generalisability of pose estimation for unseen objects and pioneers the application of RGB-D zero-shot methods in RMIS.

Teleoperated robots have enabled humans to manipulate objects in remote environments without requiring physical presence. In this paper we focus on teleoperation of a robotic arm with shared control between the robot and the operator. A model-mediated approach is used to compensate for delays in the communication channel. Position information of the operator's arm is captured and processed to compute the states of a motion prediction model before transmission over a network to be used on the robot's side, allowing for compensation of transmission delays. Model Predictive Control (MPC) and a novel goal prediction algorithm is used to follow the operator's intended motion while reducing the cognitive loads arising from collision avoidance and fine manipulation in the remote environment. We evaluate the proposed method against a baseline pure teleoperation condition with an inverse kinematic controller and observe that the proposed approach improves the overall teleoperation performance in terms of task completion time.