Gaussian Splatting (GS) is a promising scene representation for visual localization and SLAM. Recent works have explored loop closure detection via Gaussian registration, improving map consistency and accuracy. However, achieving reliable registration given two GS representations from different acquisitions remains challenging. In this paper, we propose a complete pipeline to perform the matching and registration given two GS maps. The proposed method is grounded in generating orthographic bird’s-eye views (BEVs) of optimized Gaussian models. The proposed approach leverages photometric and geometric information extracted directly from the GS to provide a trade-off of accuracy and invariance to different viewing changes (e.g., as types of GS maps, seasons, or illumination). Unlike existing 3D registration methods, which become inefficient as the number of Gaussians grows, our approach leverages 2D orthographic renders thus considerably reducing the registration complexity. Experiments on two public datasets demonstrate that our method achieves higher accuracy than several existing baselines, while also maintaining better registration results when dealing with GS maps learned by different techniques (e.g., 3DGS to LightGaussian), or GS maps presenting viewing changes such as varying illumination conditions. Source code is available at: https://gitlab.inria.fr/tangram/bev-splatreg

In this paper, we propose an object-based camera pose estimation from a single RGB image and a pre-built map of objects, represented with ellipsoidal models. We show that contrary to point correspondences, the definition of a cost function characterizing the projection of a 3D object onto a 2D object detection is not straightforward. We develop an ellipse-ellipse cost based on level sets sampling, demonstrate its nice properties for handling partially visible objects and compare its performance with other common metrics. Finally, we show that the use of a predictive uncertainty on the detected ellipses allows a fair weighting of the contribution of the correspondences which improves the computed pose. The code is released at gitlab.inria.fr/tangram/level-set-based-camera-pose-estimation.

In this work, we explore the use of objects in Simultaneous Localization and Mapping in unseen worlds and propose an object-aided system (OA-SLAM). More precisely, we show that, compared to low-level points, the major benefit of objects lies in their higher-level semantic and discriminating power. Points, on the contrary, have a better spatial localization accuracy than the generic coarse models used to represent objects (cuboid or ellipsoid). We show that combining points and objects is of great interest to address the problem of camera pose recovery. Our main contributions are: (1) we improve the relocalization ability of a SLAM system using high-level object landmarks; (2) we build an automatic system, capable of identifying, tracking and reconstructing objects with 3D ellipsoids; (3) we show that object-based localization can be used to reinitialize or resume camera tracking. Our fully automatic system allows on-the-fly object mapping and enhanced pose tracking recovery, which we think, can significantly benefit to the AR community. Our experiments show that the camera can be relocalized from viewpoints where classical methods fail. We demonstrate that this localization allows a SLAM system to continue working despite a tracking loss, which can happen frequently with an uninitiated user. Our code and test data are released at gitlab.inria.fr/tangram/oa-slam.

In this paper, we propose a method for coarse camera pose computation which is robust to viewing conditions and does not require a detailed model of the scene. This method meets the growing need of easy deployment of robotics or augmented reality applications in any environments, especially those for which no accurate 3D model nor huge amount of ground truth data are available. It exploits the ability of deep learning techniques to reliably detect objects regardless of viewing conditions. Previous works have also shown that abstracting the geometry of a scene of objects by an ellipsoid cloud allows to compute the camera pose accurately enough for various application needs. Though promising, these approaches use the ellipses fitted to the detection bounding boxes as an approximation of the imaged objects. In this paper, we go one step further and propose a learning-based method which detects improved elliptic approximations of objects which are coherent with the 3D ellipsoid in terms of perspective projection. Experiments prove that the accuracy of the computed pose significantly increases thanks to our method and is more robust to the variability of the boundaries of the detection boxes. This is achieved with very little effort in terms of training data acquisition - a few hundred calibrated images of which only three need manual object annotation.

In computer vision, estimating camera pose from correspondences between 3D geometric entities and their projections into the image is a widely investigated problem. Although most state-of-the-art methods exploit simple primitives such as points or lines, and thus require dense scene models, the emergence of very effective CNN-based object detectors in the recent years have paved the way to the use of much lighter 3D models composed solely of a few semantically relevant features. In that context, we propose a novel model-based camera pose estimation method in which the scene is modeled by a set of virtual ellipsoids. We show that 6-DoF camera pose can be determined by optimizing only the three orientation parameters, and that at least two correspondences between 3D ellipsoids and their 2D projections are necessary in practice. We validate the approach on both simulated and real environments.

We are interested in AR applications which take place in man-made GPS-denied environments, as industrial or indoor scenes. In such environments, relocalization may fail due to repeated patterns and large changes in appearance which occur even for small changes in viewpoint. We investigate in this paper a new method for relocalization which operates at the level of objects and takes advantage of the impressive progress realized in object detection. Recent works have opened the way towards object oriented reconstruction from elliptic approximation of objects detected in images. We go one step further and propose a new method for pose computation based on ellipse/ellipsoid correspondences. We consider in this paper the practical common case where an initial guess of the rotation matrix of the pose is known, for instance with an inertial sensor or from the estimation of orthogonal vanishing points. Our contributions are twofold: we prove that a closed form estimate of the translation can be computed from one ellipse-ellipsoid correspondence. The accuracy of the method is assessed on the LINEMOD database using only one correspondence. Second, we prove the effectiveness of the method on real scenes from a set of object detections generated by YOLO. A robust framework that is able to choose the best set of hypotheses is proposed and is based on an appropriate estimation of the reprojection error of ellipsoids. Globally, considering pose at the level of object allows us to avoid common failures due to repeated structures. In addition, due to the small combinatory induced by object correspondences, our method is well suited to fast rough localization even in large environments.

Given two views of the same scene, usual correspondence geometry estimation techniques exploit the well-established effectiveness of keypoint descriptors. However, such features have a hard time in poorly textured man-made environments, possibly containing repetitive patterns and/or specularities, such as industrial places. In that paper, we propose a novel method for two-view epipolar and planar geometry estimation that first aims at detecting and matching physical vertical planes frequently present in these environments, before estimating corresponding homographies. Inferred local correspondences are finally used to improve fundamental matrix estimation. The gain in precision is demonstrated on industrial and urban environments.

We introduce a novel object proposals method specific to building facades. We define new image cues that measure typical facade characteristics such as semantic, symmetry and repetitions. They are combined to generate a few facade candidates in urban environments fast. We show that our method outperforms state-of-the-art object proposals techniques for this task on the 1000 images of the Zurich Building Database. We demonstrate the interest of this procedure for augmented reality through facade recognition and camera pose initialization. In a very time-efficient pipeline we classify the candidates and match them to a facade references database using CNN-based descriptors. We prove that this approach is more robust to severe changes of viewpoint and occlusions than standard object recognition methods.

We consider the problem of camera pose estimation from a scene model obtained beforehand by a structure-from-motion (SfM) algorithm. The model is made of 3D points, each one of them being represented by its coordinates and a set of photometric descriptors such as SIFT, extracted from some of the input images of the SfM stage. Pose estimation is based on the matching of interest points from a test view with model points, using the descriptors. Descriptors having a limited invariance with respect to viewpoint changes, such an approach is likely to fail when the test view is far away from the images used to construct the model. Viewpoint simulation techniques, as ASIFT, have proved effective for wide-baseline image matching. This paper explores how these techniques can enrich a scene model by adding descriptors from simulated views, using either orthographic or pinhole virtual cameras. Viewpoint simulation permits pose estimation in situations where the approach based on the sole SIFT descriptors simply fails.

This paper presents a method for real-time augmented reality of internal liver structures during minimally invasive hepatic surgery. Vessels and tumors computed from pre-operative CT scans can be overlaid onto the laparoscopic view for surgery guidance. Compared to current methods, our method is able to locate the in-depth positions of the tumors based on partial three-dimensional liver tissue motion using a real-time biomechanical model. This model permits to properly handle the motion of internal structures even in the case of anisotropic or heterogeneous tissues, as it is the case for the liver and many anatomical structures. Experimentations conducted on phantom liver permits to measure the accuracy of the augmentation while real-time augmentation on in vivo human liver during real surgery shows the benefits of such an approach for minimally invasive surgery.

This paper focuses on the 3D shape recovery and augmented reality on elastic objects with self-occlusions handling, using only single view images. Shape recovery from a monocular video sequence is an underconstrained problem and many approaches have been proposed to enforce constraints and resolve the ambiguities. State-of-the art solutions enforce smoothness or geometric constraints, consider specific deformation properties such as inextensibility or resort to shading constraints. However, few of them can handle properly large elastic deformations. We propose in this paper a real-time method that uses a mechanical model and able to handle highly elastic objects. The problem is formulated as an energy minimization problem accounting for a non-linear elastic model constrained by external image points acquired from a monocular camera. This method prevents us from formulating restrictive assumptions and specific constraint terms in the minimization. In addition, we propose to handle self-occluded regions thanks to the ability of mechanical models to provide appropriate predictions of the shape. Our method is compared to existing techniques with experiments conducted on computer-generated and real data that show the effectiveness of recovering and augmenting 3D elastic objects. Additionally, experiments in the context of minimally invasive liver surgery are also provided and results on deformations with the presence of self-occlusions are exposed.

This article introduces a method for tracking the internal structures of the liver during robot-assisted procedures. Vascular network, tumors and cut planes, computed from pre-operative data, can be overlaid onto the laparoscopic view for image-guidance, even in the case of large motion or deformation of the organ. Compared to current methods, our method is able to precisely propagate surface motion to the internal structures. This is made possible by relying on a fast yet accurate biomechanical model of the liver combined with a robust visual tracking approach designed to properly constrain the model. Augmentation results are demonstrated on in-vivo sequences of a human liver during robotic surgery, while quantitative validation is performed on an ex-vivo porcine liver experimentation. Validation results show that our approach gives an accurate surface registration with an error of less than 6mm on the position of the tumor.

This paper proposes an efficient method to capture and augment highly elastic objects from a single view. 3D shape recovery from a monocular video sequence is an underconstrained problem and many approaches have been proposed to enforce constraints and resolve the ambiguities. State-of-the art solutions enforce smoothness or geometric constraints, consider specific deformation properties such as inextensibility or ressort to shading constraints. However, few of them can handle properly large elastic deformations. We propose in this paper a real-time method which makes use of a mechanical model and is able to handle highly elastic objects. Our method is formulated as a energy minimization problem accounting for a non-linear elastic model constrained by external image points acquired from a monocular camera. This method prevents us from formulating restrictive assumptions and specific constraint terms in the minimization. The only parameter involved in the method is the Young's modulus where we show in experiments that a rough estimate of its value is sufficient to obtain a good reconstruction. Our method is compared to existing techniques with experiments conducted on computer-generated and real data that show the effectiveness of our approach. Experiments in the context of minimally invasive liver surgery are also provided.

This paper presents a method for real-time augmentation of vascular network and tumors during minimally invasive liver surgery. Internal structures computed from pre-operative CT scans can be overlaid onto the laparoscopic view for surgery guidance. Compared to state-of-the-art methods, our method uses a real-time biomechanical model to compute a volumetric displacement field from partial three-dimensional liver surface motion. This permits to properly handle the motion of internal structures even in the case of anisotropic or heterogeneous tissues, as it is the case for the liver and many anatomical structures. Real-time augmentation results are presented on in vivo and phantom data and illustrate the benefits of such an approach for minimally invasive surgery.

We present a method for in situ modeling of polygonal scenes, using a laser rangefinder, an IMU and a camera. The main contributions of this work are a well-founded calibration procedure, a new hybrid, driftless orientation tracking method and an easy-to-use interface based on natural interactions.

Finding point correspondences between two views is generally based on the matching of local photometric descriptors. A subsequent geometric constraint ensures that the set of matching points is consistent with a realistic camera motion. Starting from a paper by Moisan and Stival, we propose an a contrario model for matching interest points based on descriptor similarity and geometric constraints. The resulting algorithm has adaptive matching thresholds and is able to detect point correspondences whose associated descriptors are not the first nearest neighbor. We also discuss the specific difficulties raised by images containing repeated patterns which are likely to introduce correspondences beyond the nearest neighbor.

We propose a shape-based variational framework to curve evolution for the segmentation of tongue contours from MRI mid-sagittal images. In particular, we first build a PCA model on tongue contours of different articulations of a reference speaker, and use it as shape priors. The parameters of the curve representation are then manipulated to minimize an objective function. The designed energy integrates both global and local image information. The global term extracts roughly the object in the whole image domain; while the local term improves precision inside a small neighborhood around the contour. Promising results and comparisons with other approaches demonstrate the efficiency of our new model.

We present Transitive Closure based visual word formation technique for obtaining robust object representations from smoothly varying multiple views. Each one of our visual words is represented by a set of feature vectors which is obtained by performing transitive closure operation on SIFT features. We also present range-reducing tree structure to speed up the transitive closure operation. The robustness of our visual word representation is demonstrated for Structure from Motion (SfM) and location identification in video images.

International audience

International audience

Being able to animate a speech production model with articulatory data would open applications in many domains. In this paper, we first consider the problem of acquiring articulatory data from non invasive image and sensor modalities: dynamic ultrasound (US) images, stereovision 3D data, electromagnetic sensors and MRI. We here especially focus on automatic registration methods which enable the fusion of the articulatory features in a common frame. We then derive articulatory parameters by fitting these features with Maeda's model. To our knowledge, it is the first attempt to derive articulatory parameters from features automatically extracted and registered between the modalities. Results prove the soundness of the approach and the reliability of the fused articulatory data.

Inverse-depth parameterization can successfully deal with the feature initialization problem in monocular simultaneous localization and mapping applications. However, it is redundant, and when multiple landmarks are initialized from the same image, it fails to enforce the ldquocommon originrdquo constraint. The authors propose two new variants that addresses both of these issues. The experimental results indicate that the proposed approach achieves a better performance at a lower computational cost.

Fundamental matrix estimation is difficult since it is often based on correspondences that are spoilt by noise and outliers. Outliers must be thrown out via robust statistics, and noise gives uncertainty. In this article we provide a closed-form formula for the uncertainty of the so-called 8 point algorithm, which is a basic tool for fundamental matrix estimation via robust methods. As an application, we modify a well established robust algorithm accordingly, leading to a new criterion to recover point correspondences under epipolar constraint, balanced by the uncertainty of the estimation. 1 Introduction and

This paper describes an integrated system for building a multiplanar model of the scene as the camera is localized on the fly. The core of this system is a robust and accurate procedure for detecting the intersection line between two planes. User cues are used to assist the system in the mapping tasks. Synthetic results and a long video demonstrate the relevance of the method.

This paper presents a probabilistic framework for computing correspondences and fundamental matrix in the structure from motion problem. Inspired by Moisan and Stival [1], we suggest using an a contrario model, which is a good answer to threshold problems in the robust filtering context. Contrary to most existing algorithms where perceptual correspondence setting and geometry evaluation are independent steps, the proposed algorithm is an all-in-one approach. We show that it is robust to repeated patterns which are usually difficult to unambiguously match and thus raise many problems in the fundamental matrix estimation.

International audience

Abstract One of the biggest obstacles to building effective augmented reality (AR) systems is the lack of accurate sensors that report the location of the user in an environment during arbitrary long periods of movements. In this paper, we present an effective hybrid approach that integrates inertial and vision‐based technologies. This work is motivated by the need to explicitly take into account the relatively poor accuracy of inertial sensors and thus to define an efficient strategy for the collaborative process between the vision‐based system and the sensor. The contributions of this papers are threefold: (i) our collaborative strategy fully integrates the sensitivity error of the sensor: the sensitivity is practically studied and is propagated into the collaborative process, especially in the matching stage (ii) we propose an original online synchronization process between the vision‐based system and the sensor. This process allows us to use the sensor only when needed. (iii) an effective AR system using this hybrid tracking is demonstrated through an e‐commerce application in unprepared environments. Copyright © 2007 John Wiley & Sons, Ltd.

This work confronts some theoretical camera models to reality and evaluates the suitability of these models for effective augmented reality (AR). It analyses what level of accuracy can be expected in real situations using a particular camera model and how robust the results are against realistic calibration errors. An experimental protocol is used that consists of taking images of a particular scene from different quality cameras mounted on a 4DOF micro-controlled device. The scene is made of a calibration target and three markers placed at different distances of the target. This protocol enables us to consider assessment criteria specific to AR as alignment error and visual impression, in addition to the classical camera positioning error.

A hybrid approach for real-time markerless tracking is presented. Robust and accurate tracking is obtained from the coupling of camera and inertial sensor data. Unlike previous approaches, we use sensor information only when the image-based system fails to track the camera. In addition, sensor errors are measured and taken into account at each step of our algorithm. Finally, we address the camera/sensor synchronization problem and propose a method to resynchronize these two devices online. We demonstrate our method in two example sequences that illustrate the behavior and benefits of the new tracking method.

Colloque avec actes et comité de lecture. internationale.

This paper addresses the registration problem for unprepared multi-planar scenes. An interactive process is proposed to obtain accurate results using only the texture information of planes. In particular, classical preparation steps (camera calibration, scene acquisition) are greatly simplified, since they are included in the on-line registration process. Results are shown on indoor and outdoor scenes. Videos are available at url http://www.loria.fr//spl tilde/gsimon/Ismar.

Realistic merging of virtual and real objects requires that the augmented patterns be correctly occluded by foreground objects. In this paper we propose a semi-automatic method for resolving occlusion in augmented reality which makes use of key-views. Once the user has outlined the occluding objects in the key-views, our system detects automatically these occluding objects in the intermediate views. A region of interest that contains the occluding objects is first computed from the outlined silhouettes. One of the main contribution of this paper is that this region takes into account the uncertainty on the computed interframe motion. Then a deformable region-based approach is used to recover the actual occluding boundary within the region of interest from this prediction.

A model registration system capable of tracking an object, the model of which is known, in an image sequence is presented. It integrates tracking, pose determination and updating of the visible features. The heart of our system is the pose computation method, which handles various features (points, lines and free-form curves) in a very robust way and is able to give a correct estimate of the pose even when tracking errors occur. The reliability of the system is shown on an augmented reality project.

We present a new approach for resolving occlusions in augmented reality. The main interest is that it does not require 3D reconstruction of the considered scene. Our idea is to use a contour based approach and to label each contour point as being "behind" or "in front of", depending on whether it is in front of or behind the virtual object. This labeling step only requires that the contours can be tracked from frame to frame. A proximity graph is then built in order to group the contours that belong to the same occluding object. Finally, we use some kind of active contours to accurately recover the mask of the occluding object.

Vocal tract X-ray image sequences are used to study articulatory phenomena and to design approximate articulatory models. The purpose of this paper is to describe an automatic tracking tool for extracting the contours of the tongue which is the most important articulator. Tracking the tongue in X ray images is an arduous task because it appears as a weak contour and above all because it is immersed among a lot of contours (teeth, palate, ...). Hence we have developed a robust algorithm that makes a snake based method and a motion based method cooperate. Significant results show the strength of the approach.

Augmented reality is the technique by which real images can be enhanced by addition of computer-generated information. Augmented reality shows great promises in fields where a simulation in situ would be either impossible, not realistic enough or too expensive. We present in this paper an augmented reality loop that uses vision tools (perspective inversion, tracking) and show how it was used to fully enrich a sequence of the bridge of Paris with a model illuminated synthetically.

The problem of reconstructing a 3D image of an object from a few number of X-ray projections is highly underdetermined. We propose a flexible method based on the regularization of the inverse linear problem with a general quadratic criterion. The minimization is performed by an iterative algorithm with a Gauss-Seidel behaviour. Due to the discrete smooth interpolation formulation, additional linear constraints are inserted, and the method is ensured to converge to the unique minimum. The application of this method is shown for 3D reconstruction of cerebral blood vessels from six projections, and the effect of various criteria is compared to the result of other algebraic methods.

In spite of the development of new imaging techniques, X-ray images still keep a prominent place to studying articulatory phenomena.Indeed, they are still unsurpassed to obtain an overall view of the moving vocal tract.However X-ray images require that articulators contours are extracted by hand which is a tedious task.This paper describes an approach towards the automatization of the tongue contour extraction.The "Snake" method introduced in computer vision to extract contours is unable alone to achieve the task.Therefore we make "Snakes" cooperate with an optical flow method applied where contours are not sufficiently isolated from spurious contours.Our experiments have shown that the tongue is tracked successfully when it is visible, and that interaction with the user remains necessary when the tongue is obscured.

Abstract Augmented reality shows great promises in fields where a simulation in situ would be impossible or too expensive. When mixing synthetic and real objects in the same animated sequence, we must be sure that the geometrical coherence as well as the photometrical coherence is ensured. One major challenge is to compute the camera viewpoint with sufficient accuracy to ensure a satisfactory composition. We especially address this point in this paper using computer vision techniques and robust statistical methods. We prove that such techniques make it possible to compute almost automatically the viewpoint for long video sequences even for bad quality images in outdoor environments. Significant results on the lighting simulation of the bridges of Paris are shown.

This paper addresses the problem of 3D surface reconstruction using image sequences. It has been shown that shape recovery from three or more occluding contours of the surface is possible given a known camera motion. Recent algorithms allow the estimation of a 3D depth map of points on extremal contours. We present our approach which is based on local approximations up to order two of the surface. Such reconstruction must then be followed by a global surface description stage. This is done by constructing a triangulation of the reconstructed points and by performing a regularisation step in order to correct perturbations which affect the reconstruction. Experiments on real data were conducted which have proved the reliability of the method.

A new algorithm for tracking formants automatically is presented. From rough formant hypotheses a regularization method is used to provide formant trajectories both close to the spectrogram edge lines and sufficiently regular. The formant hypotheses are obtained by labelling edge lines of LPC or cepstrally smoothed spectrograms in terms of formants. Speech knowledge, in the form of admissible domains for F1, F2, F3, F1 vs. F2, F2 vs. F3 and of formant level is used to obtain consistent labellings. The advantage of this algorithm is that it provides reliable formant trajectories especially in regions where a knowledge of the formant transitions give important information about the location of articulation for consonants. We present very encouraging tracking results on a corpus of sentences consisting of stops and vowels. >

This paper addresses the problem of rigid and nonpolyhedral object tracking in complex environments without a priori knowledge neither on objects nor on the camera motion. The key of our approach is to propose a robust curve based tracking algorithm founded on an iterative registration method and exploiting our rigid snake model. In order to ensure the convergence of the tracking process, we then present supervision strategies we have developed to control both the prediction step and the snake convergence. Finally, since recovering accurate contour localization is important for reconstruction task, we propose a cooperative use of the snake model and of a corner detector to finely detect piecewise curved contours.

The planar surfaces (3-D faces) of the objects of a polyhedral scene are built using a trinocular stereovision system. Before the reconstruction step and after the step of matching, a search is conducted among the set of triplets of matched 2-D faces (supposed to be the projection of 3-D faces) for the triplets which really correspond to planar 3-D surfaces. To accurately reconstruct the surface, a check is conducted in 2-D space for the classes of triplets of faces which correspond to 3-D coplanar structures. It is shown how these 3-D properties are checked in the 2-D space, using tools of projective geometry, allowing the constraint of the 3-D reconstruction in order to maintain these 3-D properties.>

In a formant tracking algorithm, the combination of both local viewpoint and global aspect of formant trajectories is difficult. The authors present an approach which incorporates local processing to build elementary tracks and an active method which generates rough formant track hypotheses and makes the resulting trajectories move towards the formant trajectories.>

Extrait de : Proceedings International Conference on Spoken Language Processing, Banff (Alberta, Canada), October 1992, pages 815-818

The strengths and the drawbacks of active contour models are described, and the absolute necessity of a criterion for assessing the solutions is pointed out. A method called snake growing, based on successive lengthenings of the snake, is proposed. The strength of this approach is that, at each stage, good convergence conditions are realized and initialization problems can be eliminated.>