Structure-from-Motion (SfM) is the task of estimating 3D structure and camera poses from images. We define Collaborative SfM (ColabSfM) as sharing distributed SfM reconstructions. Sharing maps requires estimating a joint reference frame, which is typically referred to as registration. However, there is a lack of scalable methods and training datasets for registering SfM reconstructions. In this paper, we tackle this challenge by proposing the scalable task of point cloud registration for SfM reconstructions. We find that current registration methods cannot register SfM point clouds when trained on existing datasets. To this end, we propose a SfM registration dataset generation pipeline, leveraging partial reconstructions from synthetically generated camera trajectories for each scene. Finally, we propose a simple but impactful neural refiner on top of the SotA registration method RoITr that yields significant improvements, which we call RefineRoITr. Our extensive experimental evaluation shows that our proposed pipeline and model enables ColabSfM. Code is available at https://github.com/EricssonResearch/ColabSfM

Localization is a computer vision task by which the position and orientation of a camera is determined from an image and environmental map. We propose a method for performing localization in a privacy preserving manner supporting two scenarios: first, when the image and map are held by a client who wishes to offload localization to untrusted third parties, and second, when the image and map are held separately by untrusting parties. Privacy preserving localization is necessary when the image and map are confidential, and offloading conserves on-device power and frees resources for other tasks. To accomplish this we integrate existing localization methods and secure multi-party computation (MPC), specifically garbled circuits, yielding proof-based security guarantees in contrast to existing obfuscation-based approaches which recent related work has shown vulnerable. We present two approaches to localization, a baseline data-oblivious adaptation of localization suitable for garbled circuits and our novel Single Iteration Localization. Our technique improves overall performance while maintaining confidentiality of the input image, map, and output pose at the expense of increased communication rounds but reduced computation and communication required per round. Single Iteration Localization is over two orders of magnitude faster than a straightforward application of garbled circuits to localization enabling real-world usage in Turbo the Snail, the first robot to offload localization without revealing input images, environmental map, position, or orientation to offload servers.

Abstract. Estimating the absolute pose of a camera given a set of [Formula: see text] points and their observations is known as the resectioning or Perspective-n-Point (PnP) problem. It is at the core of most computer vision applications and it can be stated as an instance of three-dimensional registration with point-line distances, making the error quadratic in the unknown pose. The PnP problem, though, is nonconvex due to the constraints associated with the rotation, and iterative algorithms may get trapped into any suboptimal solutions without notice. This work proposes an efficient certification algorithm for central and noncentral cameras that either confirms the optimality of a solution or is inconclusive. We exploit different sets of constraints for the rotation to assess their performance in terms of certification. Two of the formulations lack the Linear Independence Constraint Qualification (LICQ) while one of them has more constraints than variables. This hinders the usage of the “standard” procedure which estimates the Lagrange multipliers in closed-form. To overcome that, we formulate the certification as an eigenvalue optimization and solve it through a line-search method. Our evaluation on synthetic and real data shows that minimal formulations certify most solutions (more than [Formula: see text] on real data) whereas redundant formulations are able to certify all of them and even random problem instances. The proposed algorithm runs in microseconds for all these formulations.

Localization is a common task in computer vision by which the position and orientation of a camera is determined from an image and 3D map of the environment. We propose a demonstration of securely performing localization in a privacy preserving manner. This allows localization so that a lightweight client device, such as a mobile robot, may offload the computation to an untrusted server without revealing anything about their data. To accomplish this goal, we combine existing localization methods with secure multiparty computation (MPC), specifically garbled circuits. As such, the security guarantees of this work are simulation-based in contrast to existing obfuscation-based approaches to pose estimation for which privacy is inversely proportional to input size. We propose two approaches, a baseline data-oblivious adaptation of localization suitable for MPC and Single Iteration Localization which runs each localization step individually, improving performance at the cost of round complexity while maintaining confidentiality of the input image, map, and output pose. Single Iteration Localization is over two orders of magnitude faster than the data-oblivious approach enabling real-world usage in Turbo the Snail, the first robot to offload localization without revealing input images, environmental map, position, or orientation to offload servers.

Previous incremental estimation methods consider estimating a single line, requiring as many observers as the number of lines to be mapped. This leads to the need for having at least 4N state variables, with N being the number of lines. This paper presents the first approach for multi-line incremental estimation. Since lines are common in structured environments, we aim to exploit that structure to reduce the state space. The modeling of structured environments proposed in this paper reduces the state space to 3N + 3 and is also less susceptible to singular configurations. An assumption the previous methods make is that the camera velocity is available at all times. However, the velocity is usually retrieved from odometry, which is noisy. With this in mind, we propose coupling the camera with an Inertial Measurement Unit (IMU) and an observer cascade. A first observer retrieves the scale of the linear velocity and a second observer for the lines mapping. The stability of the entire system is analyzed. The cascade is shown to be asymptotically stable and shown to converge in experiments with simulated data.

Humans tend to build environments with structure, which consists of mainly planar surfaces. From the intersection of planar surfaces arise straight lines. Lines have more degrees of freedom than points. Thus, line-based structure-from-motion (SfM) provides more information about the environment. In this article, we present solutions for SfM using lines, namely, incremental SfM. These approaches consist of designing state observers for a camera’s dynamical visual system looking at a 3-D line. We start by presenting a model that uses spherical coordinates for representing the line’s moment vector. We show that this parameterization has singularities, and, therefore, we introduce a more suitable model that considers the line’s moment and shortest viewing ray. Concerning the observers, we present two different methodologies. The first uses a memory-less state-of-the-art framework for dynamic visual systems. Since the previous states of the robotic agent are accessible—while performing the 3-D mapping of the environment—the second approach aims at exploiting the use of memory to improve the estimation accuracy and convergence speed. The two models and the two observers are evaluated in simulation and real data, where mobile and manipulator robots are used.

We explore the possibility of using line intersection constraints for 3D scan registration. Typical 3D registration algorithms exploit point and plane correspondences, while line intersection constraints have not been used in the context of 3D scan registration before. Constraints from a match of pairs of intersecting lines in two 3D scans can be seen as two 3D line intersections, a plane correspondence, and a point correspondence. In this paper, we present minimal solvers that combine these different type of constraints: 1) three line intersections and one point match; 2) one line intersection and two point matches; 3) three line intersections and one plane match; 4) one line intersection and two plane matches; and 5) one line intersection, one point match, and one plane match. To use all the available solvers, we present a hybrid RANSAC loop. We propose a non-linear refinement technique using all the inliers obtained from the RANSAC. Vast experiments with simulated data and two real-data data-sets show that the use of these features and the combined solvers improve the accuracy. The code is available.

A reliable estimation of 3D parameters is a must for several applications like planning and control, in which is included Image-Based Visual Servoing. This control scheme depends directly on 3D parameters, e.g. depth of points, and/or depth and direction of 3D straight lines. Recently, a framework for Active Structure-from-Motion was proposed, addressing the former feature type. However, straight lines were not addressed. These are 1D objects, which allow for more robust detection, and tracking. In this work, the problem of Active Structure-from-Motion for 3D straight lines is addressed. An explicit representation of these features is presented, and a change of variables is proposed. The latter allows the dynamics of the line to respect the conditions for observability of the framework. A control law is used with the purpose of keeping the control effort reasonable, while achieving a desired convergence rate. The approach is validated first in simulation for a single line, and second using a real robot setup. The latter set of experiments are conducted first for a single line, and then for three lines.