Camera pose estimation is a fundamental task in visual localization but often relies on image retrieval or dense rendering, leading to high memory and computational cost. We propose a real-time retrieval-free camera pose estimation framework based on sparse cross-modal 2D–3D matching. Our approach learns descriptors for a compact point cloud using dense-render guidance and performs localization through projection-guided uncertainty-aware correspondence refinement, avoiding image database search and dense matching.

Most classical SLAM systems rely on the static scene assumption, which limits their applicability in real world scenarios. Recent SLAM frameworks have been proposed to simultaneously track the camera and moving objects. However they are often unable to estimate the canonical pose of the objects and exhibit a low object tracking accuracy. To solve this problem we propose TwistSLAM++, a semantic, dynamic, SLAM system that fuses stereo images and LiDAR information. Using semantic information, we track potentially moving objects and associate them to 3D object detections in LiDAR scans to obtain their pose and size. Then, we perform registration on consecutive object scans to refine object pose estimation. Finally, object scans are used to estimate the shape of the object and constrain map points to lie on the estimated surface within the bundle adjustment. We show on classical benchmarks that this fusion approach based on multimodal information improves the accuracy of object tracking.

We propose a new SLAM system that uses the semantic segmentation of objects and structures in the scene. Semantic information is relevant as it contains high level information which may make SLAM more accurate and robust. Our contribution is twofold: i) A new SLAM system based on ORB-SLAM2 that creates a semantic map made of clusters of points corresponding to objects instances and structures in the scene. ii) A modification of the classical Bundle Adjustment formulation to constrain each cluster using geometrical priors, which improves both camera localization and reconstruction and enables a better understanding of the scene. We evaluate our approach on sequences from several public datasets and show that it improves camera pose estimation with respect to state of the art.

Camera relocalization is needed in several applications such as augmented reality or robot navigation. However, it is still challenging to have a both real-time and accurate method. In this paper, we present our hybrid method combing machine learning approach and geometric approach for real-time camera relocalization from a single RGB image. We introduce our sparse feature regression forest to improve the machine learning part. In our regression forest, we propose a novel split function, that uses a whole feature vector instead of classical binary test function to improve the accuracy of 2D-3D point correspondences. Moreover, we use sparse feature extraction (SURF features) to reduce time processing. The results indicate that our method is the only real-time hybrid method (50ms per frame). We also achieve results as accurate as the best state-of-the-art methods (hybrid methods) and outperform machine learning based and sparse feature based methods.

In a co-located collaborative virtual environment, multiple users share the same physical tracked space and the same virtual workspace. When the virtual workspace is larger than the real workspace, navigation interaction techniques must be deployed to let the users explore the entire virtual environment. When a user navigates in the virtual space while remaining static in the real space, his/her position in the physical workspace and in the virtual workspace are no longer the same. Thus, in the context where each user is immersed in the virtual environment with a Head-Mounted-Display, a user can still perceive where his/her collaborators are in the virtual environment but not where they are in real world. In this paper, we propose and compare three methods to warn users about the position of collaborators in the shared physical workspace to ensure a proper cohabitation and safety of the collaborators. The frst one is based on a virtual grid shaped as a cylinder, the second one is based on a ghost representation of the user and the last one displays the physical safe-navigation space on the foor of the virtual environment. We conducted a user-study with two users wearing a Head-Mounted-Display in the context of a collaborative First-Person-Shooter game. Our three methods were compared with a condition where the physical tracked space was separated into two zones, one per user, to evaluate the impact of each condition on safety, displacement freedom and global satisfaction of users. Results suggest that the ghost avatar and the cylinder grid can be good alternatives to the separation of the tracked space.

Eyes are one of the most salient features of the human face, and the location of the pupil allows access to important information which can be used in several computer vision applications. Several commercial eye-trackers can estimate with good accuracy the pupil location, but need complex hardware specifications and a controlled user environment (high eye image resolution, good illumination, small head pose variations) making these solutions difficult to use in an arbitrary environment. In this paper, we present an approach based on Hough randomized regression trees. We demonstrate, by several evaluations on challenging public datasets that our approach is very robust to illumination, scale, eye movements and high head pose variations and yields a significant improvement compared to a wide range of state-of-the-art methods.

When dealing with real estate business, it is quite difficult for estate agents to make customers understand the potential and the volumes of free spaces. Thus, we propose an application that aims to solve these issues based on a laying out scenario in which a seller and a customer collaborate. As the roles of both users are different, we propose an asymmetric collaboration where the two users do not use the same interaction setup and do not benefit from the same interaction capabilities.

This paper studies the design and application of a novel visual attention model designed to compute user's gaze position automatically, i.e., without using a gaze-tracking system. The model we propose is specifically designed for real-time first-person exploration of 3D virtual environments. It is the first model adapted to this context which can compute in real time a continuous gaze point position instead of a set of 3D objects potentially observed by the user. To do so, contrary to previous models which use a mesh-based representation of visual objects, we introduce a representation based on surface-elements. Our model also simulates visual reflexes and the cognitive processes which take place in the brain such as the gaze behavior associated to first-person navigation in the virtual environment. Our visual attention model combines both bottom-up and top-down components to compute a continuous gaze point position on screen that hopefully matches the user's one. We conducted an experiment to study and compare the performance of our method with a state-of-the-art approach. Our results are found significantly better with sometimes more than 100 percent of accuracy gained. This suggests that computing a gaze point in a 3D virtual environment in real time is possible and is a valid approach, compared to object-based approaches. Finally, we expose different applications of our model when exploring virtual environments. We present different algorithms which can improve or adapt the visual feedback of virtual environments based on gaze information. We first propose a level-of-detail approach that heavily relies on multiple-texture sampling. We show that it is possible to use the gaze information of our visual attention model to increase visual quality where the user is looking, while maintaining a high-refresh rate. Second, we introduce the use of the visual attention model in three visual effects inspired by the human visual system namely: depth-of-field blur, camera- motions, and dynamic luminance. All these effects are computed based on the simulated gaze of the user, and are meant to improve user's sensations in future virtual reality applications.

This paper introduces a novel visual attention model to compute user's gaze position automatically, i.e. without using a gaze-tracking system. Our model is specifically designed for real-time first-person exploration of 3D virtual environments. It is the first model adapted to this context which can compute, in real-time, a continuous gaze point position instead of a set of 3D objects potentially observed by the user. To do so, contrary to previous models which use a mesh-based representation of visual objects, we introduce a representation based on surface-elements. Our model also simulates visual reflexes and the cognitive process which takes place in the brain such as the gaze behavior associated to first-person navigation in the virtual environment. Our visual attention model combines the bottom-up and top-down components to compute a continuous gaze point position on screen that hopefully matches the user's one. We have conducted an experiment to study and compare the performance of our method with a state-of-the-art approach. Our results are found significantly better with more than 100% of accuracy gained. This suggests that computing in realtime a gaze point in a 3D virtual environment is possible and is a valid approach as compared to object-based approaches.

Although compression of 3D content has been a widely studied topic, adaptive streaming of large virtual scenes is progressively becoming the standard way to share collaborative environments, due to the growing complexity of the shared data. The classical 'download and play' paradigm associated to standalone compression is leaving place to interactive refinements of regions-of-interest, provided given polygon budget and network capability. With the increasing number of applications and potential clients sharing reusable data, the traditional client / server architecture is likely to be over passed by less server-centered protocols, in order to distribute the complex tasks of identifying the selected data to be streamed. In this paper, we present the design and performances of a platform exploiting peer-to-peer transmission and show how the whole setting can be implemented using the MPEG-4 AFX standards.

The ever increasing speed of Internet connections has led to a point where it is actually possible for every end user to seamlessly share data on Internet. Peer-to-peer (P2P) networks are typical of this evolution. The goal of our paper is to show that thanks to self-adaptive assignment techniques, server-less P2P networks can efficiently deal with very large environments such as met in the geo-visualisation area. Our method takes advantage of a hierarchical and progressive data structure that describes the environment. In order to assess the global efficiency of this P2P technique, we have implemented a dedicated real time simulator. Experimentation results are presented using a hierarchical LOD model of a very large urban environment

The ever increasing speed of Internet connections has led to a point where it is actually possible for every end user to seamlessly share data on Internet. Peer-To-Peer (P2P) networks are typical of this evolution. The goal of our paper is to show that server-less P2P networks with self-adaptive assignment techniques can efficiently deal with very large environments such as met in the geovisualization domain. Our method allows adaptative view-dependent visualization thanks to a hierarchical and progressive data structure that describes the environment. In order to assess the global efficiency of this P2P technique, we have implemented a dedicated real time simulator. Experimentation results are presented using a hierarchical LOD model of a very large urban environment.