Consider the case of multiple visual sensors perceiving the same scene from different viewpoints. In order to achieve consistent visual perception, the problem of data association, in this case establishing correspondences between observed features, must be first solved. In this work, we consider multiway matching which is a specific instance of multi-sensory data association. Multiway matching refers to the problem of establishing correspondences among a set of images from noisy pairwise correspondences, typically by exploiting cycle- consistency. We propose a novel optimization-based formulation of multiway matching problem as a nonconvex low-rank matrix approximation problem. We propose two novel algorithms for numerically solving the problem at hand. The first one is an algorithm based on the Alternating Direction Method of Multipliers (ADMM). The second one is a Riemannian trust- region method on the multinomial manifold, the manifold of strictly positive stochastic matrices, equipped with the Fisher information metric. Experimental results demonstrate that the proposed methods have the state of the art performance in multiway matching while reducing the computational complexity compared to the state of the art.

Recovering 3D full-body human pose is a challenging problem with many applications. It has been successfully addressed by motion capture systems with body worn markers and multiple cameras. In this paper, we address the more challenging case of not only using a single camera but also not leveraging markers: going directly from 2D appearance to 3D geometry. Deep learning approaches have shown remarkable abilities to discriminatively learn 2D appearance features. The missing piece is how to integrate 2D, 3D, and temporal information to recover 3D geometry and account for the uncertainties arising from the discriminative model. We introduce a novel approach that treats 2D joint locations as latent variables whose uncertainty distributions are given by a deep fully convolutional neural network. The unknown 3D poses are modeled by a sparse representation and the 3D parameter estimates are realized via an Expectation-Maximization algorithm, where it is shown that the 2D joint location uncertainties can be conveniently marginalized out during inference. Extensive evaluation on benchmark datasets shows that the proposed approach achieves greater accuracy over state-of-the-art baselines. Notably, the proposed approach does not require synchronized 2D-3D data for training and is applicable to "in-the-wild" images, which is demonstrated with the MPII dataset.

Data association is one of the fundamental problems in multi-sensor systems. Most current techniques rely on pairwise data associations which can be spurious even after the employment of outlier rejection schemes. Considering multiple pairwise associations at once significantly increases accuracy and leads to consistency. In this work, we propose a fully decentralized method for globally consistent data association from pairwise data associations based on a distributed averaging scheme on the set of doubly stochastic matrices. We demonstrate the effectiveness of the proposed method using theoretical analysis and experimental evaluation.

We investigate the problem of estimating the 3D shape of an object defined by a set of 3D landmarks, given their 2D correspondences in a single image. A successful approach to alleviating the reconstruction ambiguity is the 3D deformable shape model and a sparse representation is often used to capture complex shape variability. But the model inference is still challenging due to the nonconvexity in the joint optimization of shape and viewpoint. In contrast to prior work that relies on an alternating scheme whose solution depends on initialization, we propose a convex approach to addressing this challenge and develop an efficient algorithm to solve the proposed convex program. We further propose a robust model to handle gross errors in the 2D correspondences. We demonstrate the exact recovery property of the proposed method, the advantage compared to several nonconvex baselines and the applicability to recover 3D human poses and car models from single images.

This paper addresses the challenge of 3D full-body human pose estimation from a monocular image sequence. Here, two cases are considered: (i) the image locations of the human joints are provided and (ii) the image locations of joints are unknown. In the former case, a novel approach is introduced that integrates a sparsity-driven 3D geometric prior and temporal smoothness. In the latter case, the former case is extended by treating the image locations of the joints as latent variables to take into account considerable uncertainties in 2D joint locations. A deep fully convolutional network is trained to predict the uncertainty maps of the 2D joint locations. The 3D pose estimates are realized via an Expectation-Maximization algorithm over the entire sequence, where it is shown that the 2D joint location uncertainties can be conveniently marginalized out during inference. Empirical evaluation on the Human3.6M dataset shows that the proposed approaches achieve greater 3D pose estimation accuracy over state-of-the-art baselines. Further, the proposed approach outperforms a publicly available 2D pose estimation baseline on the challenging PennAction dataset.

We propose a second order stochastic dynamical model for generic articulated objects whose state space is a Riemannian manifold naturally suggested by the articulation constraints. We derive the equations of a Riemannian Extended Kalman Filter to perform the structure estimation from an image sequence captured by a perspective camera. In order to theoretically validate our approach, we prove that the proposed model is locally weakly observable. Finally, we report quantitative results on both synthetic data and on real sequences from the CMU Mocap dataset.

The trifocal tensor, which describes the relation between projections of points and lines in three views, is a fundamental entity of geometric computer vision. In this work, we investigate a new parametrization of the trifocal tensor for calibrated cameras with non-colinear pinholes obtained from a quotient Riemannian manifold. We incorporate this formulation into state-of-the art methods for optimization on manifolds, and show, through experiments in pose averaging, that it produces a meaningful way to measure distances between trifocal tensors.

We investigate the problem of estimating the 3D shape of an object, given a set of 2D landmarks in a single image. To alleviate the reconstruction ambiguity, a widely-used approach is to confine the unknown 3D shape within a shape space built upon existing shapes. While this approach has proven to be successful in various applications, a challenging issue remains, i.e., the joint estimation of shape parameters and camera-pose parameters requires to solve a nonconvex optimization problem. The existing methods often adopt an alternating minimization scheme to locally update the parameters, and consequently the solution is sensitive to initialization. In this paper, we propose a convex formulation to address this problem and develop an efficient algorithm to solve the proposed convex program. We demonstrate the exact recovery property of the proposed method, its merits compared to alternative methods, and the applicability in human pose and car shape estimation.

In this paper, we explore the exponential map and its inverse, the logarithm map, for the group SIM(n) of similarity transformations in ℝnwhich are the composition of a rotation, a translation and a uniform scaling. We give a formula for the exponential map and we prove that it is surjective. We give an explicit formula for the case of n = 3 and show how to efficiently compute the logarithmic map. As an application, we use these algorithms to perform motion interpolation. Given a sequence of similarity transformations, we compute a sequence of logarithms, then fit a cubic spline that interpolates the logarithms and finally, we compute the interpolating curve in SIM(3).