Navigating surgical tools in the dynamic and tortuous anatomy of the lung's airways requires accurate, real-time localization of the tools with respect to the preoperative scan of the anatomy. Such localization can inform human operators or enable closed-loop control by autonomous agents, which would require accuracy not yet reported in the literature. In this paper, we introduce a deep learning architecture, called OffsetNet, to accurately localize a bronchoscope in the lung in real-time. After training on only 30 minutes of recorded camera images in conserved regions of a lung phantom, OffsetNet tracks the bronchoscope's motion on a held-out recording through these same regions at an update rate of 47 Hz and an average position error of 1.4 mm. Because this model performs poorly in less conserved regions, we augment the training dataset with simulated images from these regions. To bridge the gap between camera and simulated domains, we implement domain randomization and a generative adversarial network (GAN). After training on simulated images, OffsetNet tracks the bronchoscope's motion in less conserved regions at an average position error of 2.4 mm, which meets conservative thresholds required for successful tracking.

Tendon-driven continuum manipulators are widely used in minimally invasive surgeries, yet autonomous control has proven difficult given the uncertainty in modeling the robot's interaction with the anatomy. In this paper, we test the hypothesis that the anatomy's effect on the robotic manipulator can be approximated as a rotation of the model-based Jacobian. Two methods are described that estimate the orientation using a magnetic position sensor at the robot's distal tip, enabling task-space control in unknown, constrained environments. One method, Rα, determines the orientation of the robot's model using the measured orientation of the distal tip. In the second approach, RΔx, an unscented Kalman filter transforms the tip's measured change in position into an orientation estimate. These methods are validated on the manipulator in free space at three pre-rotated positions (α = 0°, -45°, -90°) and compared to model-based control and model-less control in an autonomous trajectory-following task through an anatomically accurate silicone phantom of a human lung. The results show that by rotating the model-based Jacobian through either of the two methods presented, feedback control successfully navigates the continuum manipulator farther into the lung phantom than static model-based control and model-less control.