It is typically challenging for visual or visual-inertial odometry systems to handle the problems of dynamic scenes and pure rotation. In this work, we design a novel visual-inertial odometry (VIO) system called RD-VIO to handle both of these two problems. First, we propose an IMU-PARSAC algorithm which can robustly detect and match keypoints in a two-stage process. In the first state, landmarks are matched with new keypoints using visual and IMU measurements. We collect statistical information from the matching and then guide the intra-keypoint matching in the second stage. Second, to handle the problem of pure rotation, we detect the motion type and adapt the deferred-triangulation technique during the data-association process. We make the pure-rotational frames into the special subframes. When solving the visual-inertial bundle adjustment, they provide additional constraints to the pure-rotational motion. We evaluate the proposed VIO system on public datasets and online comparison. Experiments show the proposed RD-VIO has obvious advantages over other methods in dynamic environments.

Achieving precise real-time localization and ensuring robustness are critical challenges in multi-user mobile AR applications. Leveraging collaborative information to augment tracking accuracy on lightweight devices and fortify overall system robustness emerges as a crucial necessity. In this paper, we propose a robust centralized collaborative rnulti-agent VI-SLAM system for mobile AR interaction and server-side efficient consistent mapping. The system deploys a lightweight VIO frontend on mobile devices for real-time tracking, and a backend running on a remote server to update multiple submaps. When overlapping areas between submaps across agents are detected, the system performs submap fusion to establish a globally consistent map. Additionally, we propose a map registration and fusion strategy based on covisibility areas for online registration and fusion in multi-agent scenarios. To improve the tracking accuracy of the frontend on agent, we introduce a strategy for updating the global map to the local map at a moderate frequency between the camera-rate pose estimation of the frontend VIO and the low-frequency global map optimization, using a tightly coupled strategy to achieve consistency of the multi-agent frontend poses estimation in the global map. The effectiveness of the proposed method is further confirmed by executing backend mapping on the server and deploying VIO frontends on multiple mobile devices for AR demostration. Additionally, we discuss the scalability of the proposed system by analyzing network traffic, synchronization frequency, and other factors at both the agent and server ends.

Abstract We propose RLP‐VIO—a robust and lightweight monocular visual‐inertial odometry system using multiplane priors. With planes extracted from the point cloud, visual‐inertial‐plane PnP uses the plane information for fast localization. Depth estimation is susceptible to degenerated motion, so the planes are expanded in a reprojection consensus‐based way robust to depth errors. For sensor fusion, our sliding‐window optimization uses a novel structureless plane‐distance error cost, which prevents the fill‐in effect that poisons the BA problem's sparsity and permits the use of a smaller sliding window while maintaining good accuracy. The total computational cost is further reduced with our modified marginalization strategy. To further improve the tracking robustness, the landmark depths are constrained using the planes during degenerated motion. The whole system is parallelized with a three‐stage pipeline. Under controlled environments, this parallelization runs deterministically and produces consistent results. The resulting VIO system is tested on widely used datasets and compared with several state‐of‐the‐art systems. Our system achieves competitive accuracy and works robustly even on long and challenging sequences. To demonstrate the effectiveness of the proposed system, we also show the AR application running on mobile devices in real‐time.

Abstract Color structure of a home scene image closely relates to the material properties of its local regions. Existing color migration methods typically fail to fully infer the correlation between the coloring of local home scene regions, leading to a local blur problem. In this paper, we propose a color migration framework for home scene images. It picks the coloring from a template image and transforms such coloring to a home scene image through a simple interaction. Our framework comprises three main parts. First, we carry out an interactive segmentation to divide an image into local regions and extract their corresponding colors. Second, we generate a matching color table by sampling the template image according to the color structure of the original home scene image. Finally, we transform colors from the matching color table to the target home scene image with the boundary transition maintained. Experimental results show that our method can effectively transform the coloring of a scene matching with the color composition of a given natural or interior scenery.

Monocular visual-inertial tracking without good initialization easily fails due to its non-linear nature. Rapid and accurate metric initialization is crucial. In this paper, we propose a novel monocular visual-inertial initialization method which can initialize the IMU states, camera poses, and scale in a rapid and robust way. To avoid mixing gravity and accelerometer bias, we propose to use the detected vertical edges to estimate a better gravity. This improves the observability to the underlying problem even without sufficient movement, so we can solve all the states crucial for a good initialization. We evaluate our approach on EuRoC dataset and compare with existing state-of-the-art methods. The experimental results demonstrate the effectiveness of the proposed method.

Although VSLAM/VISLAM has achieved great success, it is still difficult to quantitatively evaluate the localization results of different kinds of SLAM systems from the aspect of augmented reality due to the lack of an appropriate benchmark. For AR applications in practice, a variety of challenging situations (e.g., fast motion, strong rotation, serious motion blur, dynamic interference) may be easily encountered since a home user may not carefully move the AR device, and the real environment may be quite complex. In addition, the frequency of camera lost should be minimized and the recovery from the failure status should be fast and accurate for good AR experience. Existing SLAM datasets/benchmarks generally only provide the evaluation of pose accuracy and their camera motions are somehow simple and do not fit well the common cases in the mobile AR applications. With the above motivation, we build a new visual-inertial dataset as well as a series of evaluation criteria for AR. We also review the existing monocular VSLAM/VISLAM approaches with detailed analyses and comparisons. Especially, we select 8 representative monocular VSLAM/VISLAM approaches/systems and quantitatively evaluate them on our benchmark. Our dataset, sample code and corresponding evaluation tools are available at the benchmark website http://www.zjucvg.net/eval-vislam/.