Robot-assisted surgery (RAS) has raised concerns within human-computer interaction, particularly regarding the socio-material configurations of robotic systems and their impact on surgical practices. To investigate these configurations in the context of the da Vinci robotic system, we used a research-through-design approach. Through this approach, we developed six Visual RAS (ViRAS) scenarios derived from RAS observations. These scenarios represent different configurations of interacting surgical team members and the robotic system in distinct adverse RAS events. In ViRAS-guided interviews with experienced RAS surgeons, we found that ViRAS scenarios help reflect on surgical practices and the specific material and spatial properties of RAS. Our findings indicated that the team’s cognitive engagement during surgery could be improved by providing sensory augmentation to facilitate task perception and individual skill development. Through our research, we show how ViRAS scenarios, as a tool for reflection, can reveal opportunities for designing socio-material configurations in RAS and beyond.

A significant challenge in image-guided surgery is the accurate measurement task of relevant structures such as vessel segments, resection margins, or bowel lengths. While this task is an essential component of many surgeries, it involves substantial human effort and is prone to inaccuracies. In this paper, we develop a novel human-AI-based method for laparoscopic measurements utilizing stereo vision that has been guided by practicing surgeons. Based on a holistic qualitative requirements analysis, this work proposes a comprehensive measurement method, which comprises state-of-the-art machine learning architectures, such as RAFT-Stereo and YOLOv8. The developed method is assessed in various realistic experimental evaluation environments. Our results outline the potential of our method achieving high accuracies in distance measurements with errors below 1 mm. Furthermore, on-surface measurements demonstrate robustness when applied in challenging environments with textureless regions. Overall, by addressing the inherent challenges of image-guided surgery, we lay the foundation for a more robust and accurate solution for intra- and postoperative measurements, enabling more precise, safe, and efficient surgical procedures.