A popular approach to the MAP inference problem in graphical models is to minimize an upper bound obtained from a dual linear programming or Lagrangian relaxation by (block-)coordinate descent. This is also known as convex/convergent message passing; examples are max-sum diffusion and sequential tree-reweighted message passing (TRW-S). Convergence properties of these methods are currently not fully understood. They have been proved to converge to the set characterized by local consistency of active constraints, with unknown convergence rate; however, it was not clear if the iterates converge at all (to any point). We prove a stronger result (conjectured before but never proved): the iterates converge to a fixed point of the method. Moreover, we show that the algorithm terminates within $\mathcal{O}(1/\varepsilon)$ iterations. We first prove this for a version of coordinate descent applied to a general piecewise-affine convex objective. Then we show that several convex message passing methods are special cases of this method. Finally, we show that a slightly different version of coordinate descent can cycle.

We present an approach to compute bounds on the optimal value of linear programs based on constraint propagation. Given a feasible dual solution, we apply constraint propagation to the complementary slackness conditions and, if propagation succeeds to prove these conditions infeasible, the infeasibility certificate (in the sense of Farkas’ lemma) is reconstructed from the propagation history. This certificate is a dual-improving direction, which allows us to improve the bound. As constraint propagation need not always detect infeasibility of a linear inequality system, the method is not guaranteed to converge to a global solution of the linear program but only to an upper bound, whose tightness depends on the structure of the program and the used propagation method. The approach is suited for large sparse linear programs (such as LP relaxations of combinatorial optimization problems), for which the classical LP algorithms may be infeasible, if only for their super-linear space complexity. The approach can be seen as a generalization of the Virtual Arc Consistency (VAC) algorithm to bound the LP relaxation of the Weighted CSP (WCSP). We newly apply it to the LP relaxation of the Weighted Max-SAT problem, experimentally showing that the obtained bounds are often not far from optima of the relaxation and proving that they are exact for known tractable subclasses of Weighted Max-SAT.

It is well-known that for general convex optimization problems, block-coordinate descent can get stuck in poor local optima. Despite that, versions of this method known as convergent message passing are very successful to approximately solve the dual LP relaxation of the MAP inference problem in graphical models. In attempt to identify the reason why these methods often achieve good local minima, we argue that if in block-coordinate descent the set of minimizers over a variable block has multiple elements, one should choose an element from the relative interior of this set. We show that this rule is not worse than any other rule for choosing block-minimizers. Based on this observation, we develop a theoretical framework for block-coordinate descent applied to general convex problems. We illustrate this theory on convergent message-passing methods.

We show that solving linear programming (LP) relaxations of many classical NP-hard combinatorial optimization problems is as hard as solving the general LP problem. Precisely, the general LP can be reduced in linear time to the LP relaxation of each of these problems. This result poses a fundamental limitation for designing efficient algorithms to solve the LP relaxations, because finding such an algorithm might improve the complexity of best known algorithms for the general LP. Besides linear-time reductions, we show that the LP relaxations of the considered problems are P-complete under log-space reduction, therefore also hard to parallelize.

In our recent work, we showed that solving the LP relaxation of the pairwise min-sum labeling problem (also known as MAP inference in graphical models or discrete energy minimization) is not much easier than solving any linear program. Precisely, the general linear program reduces in linear time (assuming the Turing model of computation) to the LP relaxation of the min-sum labeling problem. The reduction is possible, though in quadratic time, even to the min-sum labeling problem with planar structure. Here we prove similar results for the pairwise min-sum labeling problem with attractive Potts interactions (also known as the uniform metric labeling problem).

We show that solving the LP relaxation of the min-sum labeling problem (also known as MAP inference problem in graphical models, discrete energy minimization, or valued constraint satisfaction) is not easier than solving any linear program. Precisely, every polytope is linear-time representable by a local marginal polytope and every LP can be reduced in linear time to a linear optimization (allowing infinite costs) over a local marginal polytope. The reduction can be done (though with a higher time complexity) even if the local marginal polytope is restricted to have a planar structure.

Many inference tasks in pattern recognition and artificial intelligence lead to partition functions in which addition and multiplication are abstract binary operations forming a commutative semiring. By generalizing max-sum diffusion (one of convergent message passing algorithms for approximate MAP inference in graphical models), we propose an iterative algorithm to upper bound such partition functions over commutative semirings. The iteration of the algorithm is remarkably simple: change any two factors of the partition function such that their product remains the same and their overlapping marginals become equal. In many commutative semirings, repeating this iteration for different pairs of factors converges to a fixed point when the overlapping marginals of every pair of factors coincide. We call this state marginal consistency. During that, an upper bound on the partition function monotonically decreases. This abstract algorithm unifies several existing algorithms, including max-sum diffusion and basic constraint propagation (or local consistency) algorithms in constraint programming. We further construct a hierarchy of marginal consistencies of increasingly higher levels and show than any such level can be enforced by adding identity factors of higher arity (order). Finally, we discuss instances of the framework for several semirings, including the distributive lattice and the max-sum and sum-product semirings.

We show that solving the LP relaxation of the MAP inference problem in graphical models (also known as the min-sum problem, energy minimization, or weighted constraint satisfaction) is not easier than solving any LP. More precisely, any polytope is linear-time represent able by a local marginal polytope and any LP can be reduced in linear time to a linear optimization (allowing infinite weights) over a local marginal polytope.

We present a number of contributions to the LP relaxation approach to weighted constraint satisfaction (= Gibbs energy minimization). We link this approach to many works from constraint programming, which relation has so far been ignored in machine vision and learning. While the approach has been mostly considered only for binary constraints, we generalize it to n-ary constraints in a simple and natural way. This includes a simple algorithm to minimize the LP-based upper bound, n-ary max-sum diffusion-however, we consider using other bound-optimizing algorithms as well. The diffusion iteration is tractable for a certain class of high-arity constraints represented as a black box, which is analogical to propagators for global constraints CSP. Diffusion exactly solves permuted n-ary supermodular problems. A hierarchy of gradually tighter LP relaxations is obtained simply by adding various zero constraints and coupling them in various ways to existing constraints. Zero constraints can be added incrementally, which leads to a cutting-plane algorithm. The separation problem is formulated as finding an unsatisfiable subproblem of a CSP.

LP relaxation approach to soft constraint optimisation (i.e. MAP-MRF) has been mostly considered only for binary problems. We present its generalisation to n-ary problems, including a simple algorithm to optimise the LP bound, n-ary max-sum diffusion. As applications, we show that a hierarchy of gradually tighter polyhedral relaxations of MAP-MRF is obtained by adding zero interactions. We propose a cutting plane algorithm, where cuts correspond to adding zero interactions and the separation problem to finding an unsatisfiable constraint satisfaction subproblem. Next, we show that certain high-arity interactions, e.g. certain global constraints, can be included into the framework in a principled way. Finally, we prove that n-ary max-sum diffusion finds global optimum for n-ary supermodular problems.

Inference tasks in Markov random fields (MRFs) are closely related to the constraint satisfaction problem (CSP) and its soft generalizations. In particular, MAP inference in MRF is equivalent to the weighted (maxsum) CSP. A well-known tool to tackle CSPs are arc consistency algorithms, a.k.a. relaxation labeling. A promising approach to MAP inference in MRFs is linear programming relaxation solved by sequential treereweighted message passing (TRW-S). There is a not widely known algorithm equivalent to TRW-S, max-sum diffusion, which is slower but very simple. We give two theoretical results. First, we show that arc consistency algorithms and max-sum diffusion become the same thing if formulated in an abstractalgebraic way. Thus, we argue that max-sum arc consistency algorithm or max-sum relaxation labeling is a more suitable name for max-sum diffusion. Second, we give a criterion that strictly decreases during these algorithms. It turns out that every class of equivalent problems contains a unique problem that is minimal w.r.t. this criterion.

The max-sum labeling problem, defined as maximizing a sum of binary (i.e., pairwise) functions of discrete variables, is a general NP-hard optimization problem with many applications, such as computing the MAP configuration of a Markov random field. We review a not widely known approach to the problem, developed by Ukrainian researchers Schlesinger et al. in 1976, and show how it contributes to recent results, most importantly, those on the convex combination of trees and tree-reweighted max-product. In particular, we review Schlesinger et al.'s upper bound on the max-sum criterion, its minimization by equivalent transformations, its relation to the constraint satisfaction problem, the fact that this minimization is dual to a linear programming relaxation of the original problem, and the three kinds of consistency necessary for optimality of the upper bound. We revisit problems with Boolean variables and supermodular problems. We describe two algorithms for decreasing the upper bound. We present an example application for structural image analysis.

A RANSAC-based algorithm for robust estimation of epipolar geometry from point correspondences in the possible presence of a dominant scene plane is presented. The algorithm handles scenes with (i) all points in a single plane, (ii) majority of points in a single plane and the rest off the plane, (iii) no dominant plane. It is not required to know a priori which of the cases (i)-(iii) occurs. The algorithm exploits a theorem we proved, that if five or more of seven correspondences are related by a homography then there is an epipolar geometry consistent with the seven-tuple as well as with all correspondences related by the homography. This means that a seven point sample consisting of two outliers and five inliers lying in a dominant plane produces an epipolar geometry which is wrong and yet consistent with a high number of correspondences. The theorem explains why RANSAC often fails to estimate epipolar geometry in the presence of a dominant plane. Rather surprisingly, the theorem also implies that RANSAC-based homography estimation is faster when drawing nonminimal samples of seven correspondences than minimal samples of four correspondences.

It is known that epipolar constraint can be augmented with orientation by formulating it in the oriented projective geometry. This oriented epipolar constraint requires knowing the orientations (signs of overall scales) of epipoles and fundamental matrix. The current belief is that these orientations cannot be obtained from the fundamental matrix only and that additional information is needed, typically, a single correct point correspondence. In contrary to this, we show that fundamental matrix alone encodes orientation of epipoles up to their common scale sign. We present two formulations of this fact. The algebraic formulation gives a closed formula to compute the second epipole from fundamental matrix and the first epipole. The geometric formulation is in terms of the conic formed by intersections of corresponding epipolar lines in the common image plane; we show that the epipoles always lie on different antipodal components of the spherical interpretation of this conic. Further, we show that, under mild assumptions, fundamental matrix can discriminate between two classes of mutual position of a pair of directional cameras. 1

It is well-known that epipolar geometry relating two uncalibrated images is determined by at least seven correspondences. If there are more than seven of them, their positions cannot be arbitrary if they are to be projections of any world points by any two cameras. Less than seven matches have been thought not to be constrained in any way. We show that there is a constraint even on five matches, i.e., that there exist forbidden configurations of five points in two images. The constraint is obtained by requiring orientation consistence points on the wrong side of rays are not allowed. For allowed configurations, we show that epipoles must lie in domains with piecewise-conic boundaries, and how to compute them. We present a concise algorithm deciding whether a configuration is allowed or forbidden.

Multiple projections of a scene cannot be arbitrary, the allowed configurations being given by matching constraints. This paper presents new matching constraints on multiple projections of a rigid point set by uncalibrated cameras, obtained by formulation in the oriented projective rather than projective geometry. They follow from consistency of orientations of camera rays and from the fact that the scene is the affine rather that projective space. For their non-parametric nature, we call them combinatorial. The constraints are derived in a unified theoretical framework using the theory of oriented matroids. For example, we present constraints on 4 point correspondences for 2D camera resectioning, on 3 correspondences in two 1D cameras, and on 4 correspondences in two 2D cameras.

We describe progress in automatically fitting a plane plus modelled perturbation surface model to represent architectural scenes. There are two areas of novelty. The first is a method of fitting parametrized models in which the cost function is based on a combination of disparity and gradient extrema, both computed over multiple views. The second is the use of an evaluation criteria for model selection, learnt from training examples. We demonstrate the method on reconstructions of several college scenes from multiple images.

This paper reviews two research themes. The first is wide baseline matching - establishing correspondences and cameras for images acquired from very different viewpoints. The second is automated surface reconstruction of a 3D scene from close-range images. These two themes are linked in an example application of automated architectural reconstruction from images. We demonstrate reconstructions of several university buildings from multiple photographs.

Well-known matching constraints for points and lines in muliple images are necessary but not sufficient condition for the existence of real structure and cameras, underlying the image correspondences. To obtain sufficient conditions, the following additional constraints must be imposed: positive scales, the existence of a plane at infinity not intersecting the scene, and the existence of handedness preserving cameras. We present modifications of the well-known matching constraints and also some new constraints, taking into account some of this additional knowledge. Not only conventional but also central panoramic cameras are naturally described. To achieve this, we have generalized and simplified Hartley's ch(e)irality theory by formulating it in the language of oriented projective geometry and Grassmann tensors.

This work describes a method for synthesizing correct virtual images of a real scene, described by a set of uncalibrated reference images. The approach used is rendering a projective 3D model. Projective model reconstruction and positioning a projective virtual camera are not addressed. First, the algorithm transfers the vertices of triangles in the triangulated model and then warps interiors of triangles. Hidden faces are removed by z-buffering. The algorithm is more efficient than the ray-tracing-like algorithm for virtual view synthesis by Laveau and Faugeras (1996).

Recently, much attention has been devoted to image-based scene representations. They allow one to construct an arbitrary view of a 3D scene by the interpolation (transfer) from a sparse set of real 2D (reference) images, rather than by rendering an explicit 3D model. While many authors address mainly the purely geometrical aspect of the task, we focus on the problem of how to select the optimal set of reference views. Selection of reference views from a dense set of real primary views is posed as a selection and fitting of parametric models. The selected set must minimize a weighted sum of the number of reference views and the total fit error. We propose two different algorithms for solving this optimization problem. The experimental results on synthetic and real data indicate the feasibility of the approach for 1-DOF camera movement. We discuss the possibility to extend one of the algorithms for more general case.

Presents a new approach to rendering arbitrary views of real-world 3D objects of complex shapes. We propose to represent an object by a sparse set of corresponding 2D views, and to construct any other view as a combination of these reference views. We show that this combination can be linear, assuming proximity of the views, and we suggest how the visibility of constructed points can be determined. Our approach makes it possible to avoid difficult 3D reconstruction, assuming only rendering is required. Moreover, almost no calibration of views is needed. We present preliminary results on real objects, indicating that the approach is feasible.>