We present ESLAM, an efficient implicit neural representation method for Simultaneous Localization and Mapping (SLAM). ESLAM reads RGB-D frames with unknown camera poses in a sequential manner and incrementally reconstructs the scene representation while estimating the current camera position in the scene. We incorporate the latest advances in Neural Radiance Fields (NeRF) into a SLAM system, resulting in an efficient and accurate dense visual SLAM method. Our scene representation consists of multi-scale axis-aligned perpendicular feature planes and shallow decoders that, for each point in the continuous space, decode the interpolated features into Truncated Signed Distance Field (TSDF) and RGB values. Our extensive experiments on three standard datasets, Replica, ScanNet, and TUM RGB-D show that ESLAM improves the accuracy of 3D reconstruction and camera localization of state-of-the-art dense visual SLAM methods by more than 50%, while it runs up to ×10 faster and does not require any pre-training. Project page: https://www.idiap.ch/paper/eslam.

We present GeoNeRF, a generalizable photorealistic novel view synthesis method based on neural radiance fields. Our approach consists of two main stages: a ge-ometry reasoner and a renderer. To render a novel view, the geometry reasoner first constructs cascaded cost volumes for each nearby source view. Then, using a Transformer- based attention mechanism and the cascaded cost volumes, the renderer infers geometry and appearance, and ren-ders detailed images via classical volume rendering techniques. This architecture, in particular, allows sophis-ticated occlusion reasoning, gathering information from consistent source views. Moreover, our method can eas-ily be fine-tuned on a single scene, and renders com-petitive results with per-scene optimized neural rendering methods with a fraction of computational cost. Ex-periments show that GeoNeRF outperforms state-of-the- art generalizable neural rendering models on various syn-thetic and real datasets. Lastly, with a slight modification to the geometry reasoner, we also propose an alter-native model that adapts to RGBD images. This model di-rectly exploits the depth information often available thanks to depth sensors. The implementation code is available at https://www.idiap.ch/paper/geonerf.

We present DepthInSpace, a self-supervised deep-learning method for depth estimation using a structured-light camera. The design of this method is motivated by the commercial use case of embedded depth sensors in nowadays smartphones. We first propose to use estimated optical flow from ambient information of multiple video frames as a complementary guide for training a single-frame depth estimation network, helping to preserve edges and reduce over-smoothing issues. Utilizing optical flow, we also propose to fuse the data of multiple video frames to get a more accurate depth map. In particular, fused depth maps are more robust in occluded areas and incur less in flying pixels artifacts. We finally demonstrate that these more precise fused depth maps can be used as self-supervision for fine-tuning a single-frame depth estimation network to improve its performance. Our models’ effectiveness is evaluated and compared with state-of-the-art models on both synthetic and our newly introduced real datasets. The implementation code, training procedure, and both synthetic and captured real datasets are available at https://www.idiap.ch/paper/depthinspace.

The colorization of gray-scale images has always been a challenging task in computer vision. Recently, novel approaches have been introduced for unsupervised image translation between two domains using Generative Adversarial Networks (GANs). Since one can consider the gray-scale and colorful images as two separate domains, we propose a two-stage cycle-consistent network architecture to produce convincible images. First, an intermediate image is generated with a relatively uncomplicated objective function at the output. Next, at the second stage, the intermediate image is enhanced via a residual network structure with a more complicated objective function. Furthermore, by employing two inverse networks, a cycle-consistent architecture is formed at both stages. The proposed model is trained on the ImageNet dataset, and the achieved outcomes demonstrate exceptional performance comparing with the state-of-the-art models.