Surgical automation requires precise guidance and understanding of the scene. Current methods in the literature rely on bulky depth cameras to create maps of the anatomy; however, this does not translate well to space-limited clinical applications. Monocular cameras are small and allow minimally invasive surgeries in tight spaces, but additional processing is required to generate 3D scene understanding. We propose a 3D mapping pipeline that uses only RGB images to create segmented point clouds of the target anatomy. To ensure the most accurate reconstruction, we compare different structure from motion algorithms’ performance on mapping the central airway obstructions, and test the pipeline on a downstream task of tumor resection. In several metrics, including post-procedure percentage tissue charring, our pipeline performs comparably to RGB-D cameras and, in some cases, even surpasses their downstream task performance. These promising results demonstrate that automation guidance can be achieved in minimally invasive procedures with monocular cameras. This study is a step toward the complete autonomy of surgical robots.

Femoral artery access is essential for numerous clinical procedures, including diagnostic angiography, therapeutic catheterization, and emergency interventions. Despite its critical role, successful vascular access remains challenging due to anatomical variability, overlying adipose tissue, and the need for precise ultrasound (US) guidance. Needle placement errors can result in severe complications, thereby limiting the procedure to highly skilled clinicians operating in controlled hospital environments. While robotic systems have shown promise in addressing these challenges through autonomous scanning and vessel reconstruction, clinical translation remains limited due to reliance on simplified phantom models that fail to capture human anatomical complexity. In this work, we present a method for autonomous robotic US scanning of bifurcated femoral arteries, and validate it on five vascular phantoms created from real patient computed tomography (CT) data. Additionally, we introduce a video-based deep learning US segmentation network tailored for vascular imaging, enabling improved 3D arterial reconstruction. The proposed network achieves a Dice score of 89.21% and an Intersection over Union of 80.54% on a new vascular dataset. The reconstructed artery centerline is evaluated against ground truth CT data, showing an average L2error of 0.91±0.70 mm, with an average Hausdorff distance of 4.36±1.11mm. This study is the first to validate an autonomous robotic system for US scanning of the femoral artery on a diverse set of patient-specific phantoms, introducing a more advanced framework for evaluating robotic performance in vascular imaging and intervention.

Percutaneous needle insertions are commonly performed for diagnostic and therapeutic purposes as an effective alternative to more invasive surgical procedures. However, the outcome of needle-based approaches relies heavily on the accuracy of needle placement, which remains a challenge even with robot assistance and medical imaging guidance due to needle deflection caused by contact with soft tissues. In this paper, we present a novel mechanics-based 2D bevel-tip needle model that can account for the effect of nonlinear strain-dependent behavior of biological soft tissues under compression. Real-time finite element simulation allows multiple control inputs along the length of the needle with full three-degree-of-freedom (DOF) planar needle motions. Cross-validation studies using custom-designed multi-layer tissue phantoms as well as heterogeneous chicken breast tissues result in less than 1mm in-plane errors for insertions reaching depths of up to 61 mm, demonstrating the validity and generalizability of the proposed method.

Skin segmentation plays an important role in a wide variety of biomedical image processing applications, such as skin cancer identification, skin lesion detection, and wound isolation. However, contemporary research has been mainly based on facial and hand skin datasets, with no other body regions considered for skin pixels sampling. Segmenting skin specifically in the abdominal region can aid in robotic abdominal surgeries and treatment procedures, such as robot-assisted laparoscopic surgeries and abdominal ultrasounds. A robust and highly accurate abdominal skin detection technique thus becomes imperative. To this end, we compiled a novel dataset of 1,400 segmented abdominal pictures and adapted and compared four abdominal skin segmentation techniques: one based on thresholding and three deep learning techniques, namely a fully connected neural network for pixel-level classification, and two convolution-based networks, U-Net and Mask-RCNN. We show that the U-Net model outperforms the other segmentation techniques, resulting in a pixel-to-pixel mean cross-validation accuracy of 95.51% on our Abdominal dataset. The incorporation of the Abdominal dataset in the training helped improve the abdominal skin segmentation accuracy by 10.19%. The U-Net model proved to be computationally the fastest, enabling real time skin segmentation with a processing rate of 37 frames per second.