Continuum Dexterous Manipulators (CDMs) are well-suited tools for minimally invasive surgery due to their inherent dexterity and reachability. Nonetheless, their flexible structure and non-linear curvature pose significant challenges for shape-based feedback control. The use of Fiber Bragg Grating (FBG) sensors for shape sensing has shown great potential in estimating the CDM’s tip position and subsequently reconstructing the shape using optimization algorithms. This optimization, however, is under-constrained and may be ill-posed for complex shapes, falling into local minima. In this work, we introduce a novel method capable of directly estimating a CDM’s shape from FBG sensor wavelengths using a deep neural network. In addition, we propose the integration of uncertainty estimation to address the critical issue of uncertainty in neural network predictions. Neural network predictions are unreliable when the input sample is outside the training distribution or corrupted by noise. Recognizing such deviations is crucial when integrating neural networks within surgical robotics, as inaccurate estimations can pose serious risks to the patient. We present a robust method that not only improves the precision upon existing techniques for FBG-based shape estimation but also incorporates a mechanism to quantify the models’ confidence through uncertainty estimation. We validate the uncertainty estimation through extensive experiments, demonstrating its effectiveness and reliability on out-of-distribution (OOD) data, adding an additional layer of safety and precision to minimally invasive surgical robotics.

Arthroscopy is a minimally invasive orthopedic procedure commonly used to treat joints such as the shoulder, hip, or knee. A major difficulty for surgeons in arthroscopic procedures is the simultaneous coordination of the surgical tools used for manipulation and the arthroscopic camera. To further complicate this task, the narrow space where arthroscopic procedures are performed limits the ability to move the arthroscope inside the patient’s body, restricting the field of view. In this work, to overcome these limitations, we introduce ARthroNeRF, a novel framework that combines Neural Radiance Fields (NeRF) and Augmented Reality (AR). This framework allows for the generation of synthetic viewpoints from the perspective of surgical tools without the need for an additional camera. To evaluate the feasibility of the proposed framework, we conducted a user study with 18 participants. In this study, participants were tasked to touch hidden targets assisted by a synthetic view generated from the perspective of a surgical tool. The results of this study demonstrate that ARthroNeRF provides accurate supplementary visual information and suggest that ARthroNeRF has the potential to streamline the learning process in arthroscopic surgery. In addition, we built a system capable of presenting the reconstructed scenes using the Microsoft HoloLens 2. The incorporation of AR, overlaying synthesized images alongside the original arthroscopic footage and within the surgeon’s visual field, represents a viable alternative to enhance perception in spatially constrained scenarios.

Using Field Programmable Gate Arrays (FPGAs) as implementation platform for systems-on-chip (SoC) has become quite popular. Typically, the software part of the system functionality is executed on a soft-core processor. Debugging such systems becomes more difficult than standard SoCs since regular debugging facilities are not always available for the processor cores and also additional hardware problems can overlap with software bugs. Thus, it is interesting to provide a general debugging framework that can help to identify SW and HW problems. In this contribution, we use the readback feature of modern FPGAs to implement such a general framework while at the same time minimizing the additional HW resources required for the debugging. We interface our debugging facilities with a full featured development environment such that the user can work at a very high level of abstraction.