Efficient and high-fidelity reconstruction of deformable surgical scenes is a critical yet challenging task. Building on recent advancements in 3D Gaussian splatting, current methods have seen significant improvements in both reconstruction quality and rendering speed. However, two major limitations remain: (1) difficulty in handling irreversible dynamic changes, such as tissue shearing, which are common in surgical scenes; and (2) the lack of hierarchical modeling for surgical scene deformation, which reduces rendering speed. To address these challenges, we introduce EH-SurGS, an efficient and high-fidelity reconstruction algorithm for deformable surgical scenes. We propose a deformation modeling approach that incorporates the life cycle of 3D Gaussians, effectively capturing both regular and irreversible deformations, thus enhancing reconstruction quality. Additionally, we present an adaptive motion hierarchy strategy that distinguishes between static and deformable regions within the surgical scene. This strategy reduces the number of 3D Gaussians passing through the deformation field, thereby improving rendering speed. Extensive experiments on public datasets captured with static endoscopes demonstrate that our method surpasses existing state-of-the-art approaches in both reconstruction quality and rendering speed. Ablation studies further validate the effectiveness and necessity of our proposed components. We will open-source our code upon acceptance of the paper.

Reconstructing deformable soft tissues from endoscopic videos is a critical yet challenging task. Leveraging depth priors, deformable implicit neural representations have seen significant advancements in this field. However, depth priors from pre-trained depth estimation models are often coarse, and inaccurate depth supervision can severely impair the performance of these neural networks. Moreover, existing methods overlook local similarities in input sequences, which restricts their effectiveness in capturing local details and tissue deformations. In this paper, we introduce UW-DNeRF, a novel approach utilizing neural radiance fields for high-quality reconstruction of deformable tissues. We propose an uncertainty-guided depth supervision strategy to mitigate the impact of inaccurate depth information. This strategy relaxes hard depth constraints and unlocks the potential of implicit neural representations. In addition, we design a local window-based information sharing scheme. This scheme employs local window and keyframe deformation networks to construct deformations with local awareness and enhances the model's ability to capture fine details. We demonstrate the superiority of our method over state-of-the-art approaches on synthetic and in vivo endoscopic datasets. Code is available at: https://github.com/IRMVLab/UW-DNeRF.

Building a self-model for robots, enabling them to simulate their physical selves and predict future states without direct interaction with the physical world, is crucial for robot motion planning and control. Existing self-modeling methods primarily focus on rigid robots and typically require significant time, effort, and resources to gather training data. In this study, we introduce SoftNeRF, a self-supervised visual self-model designed for soft robots. We use a hybrid neural shape representation based on the Signed Distance Function (SDF) to capture both the geometry and complex nonlinear motion of soft robots. By leveraging differentiable rendering, our method learns a self-model from readily available RGB images, similar to how humans understand their physical state through reflection. To improve training efficiency and model accuracy, we propose an error-guided adaptive sampling strategy. SoftNeRF can serve as a plug-in for various downstream tasks, even when trained with data unrelated to those tasks. We demonstrate SoftNeRF’s ability to support shape prediction and motion planning for robots in both simulated and real-world environments. Furthermore, SoftNeRF excels in detecting and recovering from damage, thereby enhancing machine resilience. Code is available at: https://github.com/irmvlab/soft-nerf.

Estimating camera motion and continuously reconstructing dense scenes in deformable environments presents a complex and open challenge. Many existing approaches tend to rely on assumptions about the scene’s topology or the nature of deformable motion. However, these assumptions do not hold true in medical endoscopy applications. To address these challenges, we introduce DDS-SLAM, a novel dense deformable semantic neural SLAM that achieves accurate camera tracking, continuous dense scene reconstruction, and high-quality image rendering in deformable scenes. First, we propose a novel hybrid neural scene representation method capable of capturing both natural and artificial deformations. Additionally, by leveraging the 2D semantic information of the scene, we introduce a semantic loss function based on semantic distance fields. This approach guides network optimization at a higher level, thereby enhancing system performance. Furthermore, we validate our method through a series of experiments conducted on several representative medical datasets, demonstrating its superiority over other state-of-the-art approaches. The code is available at: https://github.com/IRMVLab/DDS-SLAM.