Learning image representations that capture rich semantic relationships remains a significant challenge. Existing approaches are either contrastive, lacking robust theoretical guarantees, or struggle to effectively represent the partial orders inherent to structured visual-semantic data. In this paper, we introduce a nuclear norm-based loss function, grounded in the same information theoretic principles that have proved effective in self-supervised learning. We present a theoretical characterization of this loss, demonstrating that, in addition to promoting class orthogonality, it encodes the spectral geometry of the data within a subspace lattice. This geometric representation allows us to associate logical propositions with subspaces, ensuring that our learned representations adhere to a predefined symbolic structure.

The precise estimation of camera poses within large camera networks is a foundational problem in computer vision and robotics, with broad applications spanning autonomous navigation, surveillance, and augmented reality. In this paper, we introduce a novel methodology that extends state-of-the-art Pose Graph Optimization (PGO) techniques. Departing from the conventional PGO paradigm, which primarily relies on camera-camera edges, our approach centers on the introduction of a dynamic element - any rigid object free to move in the scene - whose pose can be reliably inferred from a single image. Specifically, we consider the bipartite graph encompassing cameras, object poses evolving dynamically, and camera-object relative transformations at each time step. This shift not only offers a solution to the challenges encountered in directly estimating relative poses between cameras, particularly in adverse environments, but also leverages the inclusion of numerous object poses to ameliorate and integrate errors, resulting in accurate camera pose estimates. Though our framework retains compatibility with traditional PGO solvers, its efficacy benefits from a custom-tailored optimization scheme. To this end, we introduce an iterative primal-dual algorithm, capable of handling large graphs. Empirical benchmarks, conducted on a new dataset of simulated indoor environments, substantiate the efficacy and efficiency of our approach.

Recent research in representation learning has shown that hierarchical data lends itself to low-dimensional and highly informative representations in hyperbolic space. However, even if hyperbolic embeddings have gathered attention in image recognition, their optimization is prone to numerical hurdles. Further, it remains unclear which applications stand to benefit the most from the implicit bias imposed by hyperbolicity, when compared to traditional Euclidean features. In this paper, we focus on prototypical hyperbolic neural networks. In particular, the tendency of hyperbolic embeddings to converge to the boundary of the Poincaré ball in high dimensions and the effect this has on few-shot classification. We show that the best few-shot results are attained for hyperbolic embeddings at a common hyperbolic radius. In contrast to prior benchmark results, we demonstrate that better performance can be achieved by a fixed-radius encoder equipped with the Euclidean metric, regardless of the embedding dimension.

A cornerstone of geometric reconstruction, rotation averaging seeks the set of absolute rotations that optimally explains a set of measured relative orientations between them. In spite of being an integral part of bundle adjustment and structure-from-motion, averaging rotations is both a non-convex and high-dimensional optimization problem. In this paper, we address it from a maximum likelihood estimation standpoint and make a twofold contribution. Firstly, we set forth a novel initialization-free primal-dual method which we show empirically to converge to the global optimum. Further, we derive what is to our knowledge, the first optimal closed-form solution for rotation averaging in cycle graphs and contextualize this result within spectral graph theory. Our proposed methods achieve a significant gain both in precision and performance.

Pose Graph Optimization (PGO) is an important problem in Computer Vision, particularly in motion estimation, whose objective consists of finding the rigid transformations that achieve the best global alignment of visual data on a common reference frame. The vast majority of PGO approaches rely on iterative techniques which refine an initial estimate until convergence is achieved. On the other hand, recent works have identified a global constraint which has cast this problem into the matrix completion domain. The success which both these formulations have had in computing accurate solutions efficiently has been overshadowed by large-scale industrial applications such as autonomous flight, self-driving cars and smart-cities, where it is necessary to fuse numerous images covering large areas but where each one of them has few pairwise observations. We propose a highly efficient algorithm to solve PGO which leverages the sparsity of the data by combining the Krylov- Schur method for spectral decomposition with Cholesky LDL factorization. Our method allows for high scalability, low computational cost and high precision, simultaneously.

Recently, conversational systems have seen a significant rise in demand due to modern commercial applications using systems such as Amazon's Alexa, Apple's Siri, Microsoft's Cortana and Google Assistant. The research on multimodal chatbots is a widely underexplored area, where users and the conversational agent communicate by natural language and visual data. Conversational agents are now becoming a commodity as a number of companies push for this technology. The wide use of these conversational agents exposes the many challenges in achieving more natural, human-like, and engaging conversational agents. The research community is actively addressing several of these challenges: how are visual and text data related in user utterances? How to interpret the user intent? How to encode multimodal dialog status? What are the ethical and legal aspects of conversational AI? The Multimodal Conversational AI workshop will be a forum where researchers and practitioners share their experiences and brainstorm about success and failures in the topic. It will also promote collaboration to strengthen the conversational AI community at ACM Multimedia.

Finding good representations for videos is becoming increasingly more important to enable an efficient analysis and comparison, with potential applications in sports, surveillance, news, or web services. This paper proposes a new representation of videos based on human pose. Rather than looking at conventional features, our method relies only on human pose detections to characterize the video. This approach provides a powerful tool for the efficient analysis of videos of human activities, particularly for video summarization and retrieval. We evaluate the proposed representation on the following tasks: 1) computing video statistics, such as the main poses and viewpoint preferences; 2) partitioning videos into a collection of short clips that will compose the video summary; and 3) retrieving frames or scenes with specific poses from videos. Results show that the proposed approach is able to successfully perform these tasks.

We propose a method to reconstruct and cluster incomplete high-dimensional data lying in a union of low-dimensional subspaces. Exploring the sparse representation model, we jointly estimate the missing data while imposing the intrinsic subspace structure. Although we have a non-convex problem, we propose an algorithm robust to initialization. Extensive experiments with synthetic and real data show that our approach leads to significant improvements in the reconstruction and segmentation, outperforming current state of the art for both low and high-rank data.

Many computer vision problems are formulated as the optimization of a cost function. This approach faces two main challenges: designing a cost function with a local optimum at an acceptable solution, and developing an efficient numerical method to search for this optimum. While designing such functions is feasible in the noiseless case, the stability and location of local optima are mostly unknown under noise, occlusion, or missing data. In practice, this can result in undesirable local optima or not having a local optimum in the expected place. On the other hand, numerical optimization algorithms in high-dimensional spaces are typically local and often rely on expensive first or second order information to guide the search. To overcome these limitations, we propose Discriminative Optimization (DO), a method that learns search directions from data without the need of a cost function. DO explicitly learns a sequence of updates in the search space that leads to stationary points that correspond to the desired solutions. We provide a formal analysis of DO and illustrate its benefits in the problem of 3D registration, camera pose estimation, and image denoising. We show that DO outperformed or matched state-of-the-art algorithms in terms of accuracy, robustness, and computational efficiency.

Rigid Point Cloud Registration (PCReg) refers to the problem of finding the rigid transformation between two sets of point clouds. This problem is particularly important due to the advances in new 3D sensing hardware, and it is challenging because neither the correspondence nor the transformation parameters are known. Traditional local PCReg methods (e.g., ICP) rely on local optimization algorithms, which can get trapped in bad local minima in the presence of noise, outliers, bad initializations, etc. To alleviate these issues, this paper proposes Inverse Composition Discriminative Optimization (ICDO), an extension of Discriminative Optimization (DO), which learns a sequence of update steps from synthetic training data that search the parameter space for an improved solution. Unlike DO, ICDO is object-independent and generalizes even to unseen shapes. We evaluated ICDO on both synthetic and real data, and show that ICDO can match the speed and outperform the accuracy of state-of-the-art PCReg algorithms.

While many approaches have been proposed to estimate and remove blur in a photo, few efforts were made to have an algorithm automatically understand the blur desirability: whether the blur is desired or not, and how it affects the quality of the photo. Such a task not only relies on low-level visual features to identify blurry regions, but also requires high-level understanding of the image content as well as user intent during photo capture. In this paper, we propose a unified framework to estimate a spatially-varying blur map and understand its desirability in terms of image quality at the same time. In particular, we use a dilated fully convolutional neural network with pyramid pooling and boundary refinement layers to generate high-quality blur response maps. If blur exists, we classify its desirability to three levels ranging from good to bad, by distilling high-level semantics and learning an attention map to adaptively localize the important content in the image. The whole framework is end-to-end jointly trained with both supervisions of pixel-wise blur responses and image-wise blur desirability levels. Considering the limitations of existing image blur datasets, we collected a new large-scale dataset with both annotations to facilitate training. The proposed methods are extensively evaluated on two datasets and demonstrate state-of-the-art performance on both tasks.

While domain adaptation has been actively researched, most algorithms focus on the single-source-single-target adaptation setting. In this paper we propose new generalization bounds and algorithms under both classification and regression settings for unsupervised multiple source domain adaptation. Our theoretical analysis naturally leads to an efficient learning strategy using adversarial neural networks: we show how to interpret it as learning feature representations that are invariant to the multiple domain shifts while still being discriminative for the learning task. To this end, we propose multisource domain adversarial networks (MDAN) that approach domain adaptation by optimizing task-adaptive generalization bounds. To demonstrate the effectiveness of MDAN, we conduct extensive experiments showing superior adaptation performance on both classification and regression problems: sentiment analysis, digit classification, and vehicle counting.

In this paper, we aim to monitor the flow of people in large public infrastructures. We propose an unsupervised methodology to cluster people flow patterns into the most typical and meaningful configurations. By processing 3-D images from a network of depth cameras, we build a descriptor for the flow pattern. We define a data-irregularity measure that assesses how well each descriptor fits a data model. This allows us to rank flow patterns from highly distinctive (outliers) to very common ones. By discarding outliers, we obtain more reliable key configurations (classes). Synthetic experiments show that the proposed method is superior to standard clustering methods. We applied it in an operational scenario during 14 days in the X-ray screening area of an international airport. Results show that our methodology is able to successfully summarize the representative patterns for such a long observation period, providing relevant information for airport management. Beyond regular flows, our method identifies a set of rare events corresponding to uncommon activities (cleaning, special security, and circulating staff).

Many computer vision problems are formulated as the optimization of a cost function. This approach faces two main challenges: (1) designing a cost function with a local optimum at an acceptable solution, and (2) developing an efficient numerical method to search for one (or multiple) of these local optima. While designing such functions is feasible in the noiseless case, the stability and location of local optima are mostly unknown under noise, occlusion, or missing data. In practice, this can result in undesirable local optima or not having a local optimum in the expected place. On the other hand, numerical optimization algorithms in high-dimensional spaces are typically local and often rely on expensive first or second order information to guide the search. To overcome these limitations, this paper proposes Discriminative Optimization (DO), a method that learns search directions from data without the need of a cost function. Specifically, DO explicitly learns a sequence of updates in the search space that leads to stationary points that correspond to desired solutions. We provide a formal analysis of DO and illustrate its benefits in the problem of 2D and 3D point cloud registration both in synthetic and range-scan data. We show that DO outperforms state-of-the-art algorithms by a large margin in terms of accuracy, robustness to perturbations, and computational efficiency.

In this paper, we develop deep spatio-temporal neural networks to sequentially count vehicles from low quality videos captured by city cameras (citycams). Citycam videos have low resolution, low frame rate, high occlusion and large perspective, making most existing methods lose their efficacy. To overcome limitations of existing methods and incorporate the temporal information of traffic video, we design a novel FCN-rLSTM network to jointly estimate vehicle density and vehicle count by connecting fully convolutional neural networks (FCN) with long short term memory networks (LSTM) in a residual learning fashion. Such design leverages the strengths of FCN for pixel-level prediction and the strengths of LSTM for learning complex temporal dynamics. The residual learning connection reformulates the vehicle count regression as learning residual functions with reference to the sum of densities in each frame, which significantly accelerates the training of networks. To preserve feature map resolution, we propose a Hyper-Atrous combination to integrate atrous convolution in FCN and combine feature maps of different convolution layers. FCN-rLSTM enables refined feature representation and a novel end-to-end trainable mapping from pixels to vehicle count. We extensively evaluated the proposed method on different counting tasks with three datasets, with experimental results demonstrating their effectiveness and robustness. In particular, FCN-rLSTM reduces the mean absolute error (MAE) from 5.31 to 4.21 on TRANCOS; and reduces the MAE from 2.74 to 1.53 on WebCamT. Training process is accelerated by 5 times on average.

Object detection has been a long standing problem in computer vision, and state-of-the-art approaches rely on the use of sophisticated features and/or classifiers. However, these learning-based approaches heavily depend on the quality and quantity of labeled data, and do not generalize well to extreme poses or textureless objects. In this work, we explore the use of 3D shape models to detect objects in videos in an unsupervised manner. We call this problem Motion from Structure (MfS): given a set of point trajectories and a 3D model of the object of interest, find a subset of trajectories that correspond to the 3D model and estimate its alignment (i.e., compute the motion matrix). MfS is related to Structure from Motion (SfM) and motion segmentation problems: unlike SfM, the structure of the object is known but the correspondence between the trajectories and the object is unknown, unlike motion segmentation, the MfS problem incorporates 3D structure, providing robustness to tracking mismatches and outliers. Experiments illustrate how our MfS algorithm outperforms alternative approaches in both synthetic data and real videos extracted from YouTube.

Due to economic and environmental issues, bicycles have been regaining their significance as a transportation vehicle in urban scenarios. To further drive this desirable trend, policy makers must have the tools to access current bicycle infrastructures and road safety concerns. Fundamental for this assessment is a deeper understanding of how cyclists use current infrastructures, if the cycling experience results in stressful events, and the conditions of the current infrastructure. We here introduce a new platform, SMARTcycling, that, by taking advantage of the mobile power available to a smartphone, captures and stores data from several sensors, namely an action camera, a cardio signal acquisition belt, and smartphone's Global Positioning System (GPS) coordinates. The data is further processed and, through visual cues, we access the cyclist driving events and road condition cues. SMARTcycling also detects the cyclist stress using the electrocardiograms (ECG) from the belt. We further contribute by making available a dataset containing the sensors data from 10 paths over two cities in Portugal. On this dataset, we show our initial promising results on event detection, road condition identification and stress assessment.

This paper presents a hybrid method for single-source localization in wireless sensor networks, fusing noisy range measurements with angular information extracted from video. Although recent works found in the literature explore hybrid schemes, these include several cumbersome assumptions. We develop and test, both numerically and experimentally, a hybrid localization algorithm which surpasses the limitations of previous fusing approaches. The proposed method (FLORIS) is based on a nonconvex least-squares joint formulation, for which a tight convex relaxation is applied to obtain a semidefinite program. Numerical simulations show that FLORIS has comparable performance to state-of-the-art methods, even outperforming them in some scenarios. Real experiments show that FLORIS is feasible in practical application scenarios, achieving very good accuracy and robustness. Importantly, coverage requirements for the infrastructure in a given area are more flexible than resorting to a single type of sensor, which may simplify practical deployments.

Global localization is a widely studied problem, and in essence corresponds to the online robot pose estimation based on a given map with landmarks, an odometry model, and real robot sensory observations and motion. In most approaches, the map provides the position of visible objects, which are then recognized to provide the robot pose estimation. Such object recognition with noisy sensory data is challenging. In this paper, we present an effective global localization technique using soft 3D object recognition to estimate the pose with respect to the landmarks in the given map. A depth sensor acquires a partial view for each observed object, from which our algorithm extracts the robot pose relative to the objects, based on a library of 3D Partial View Heat Kernel descriptors. Our approach departs from methods that require classification and registration against complete 3D models, which are prone to errors due to noisy sensory data and object misclassifications in the recognition stage. We experimentally validate our method in different robot paths with different common 3D environment objects. We also show the improvement of our method compared to when the partial view information is not used.

In the last few years, image classification has become an incredibly active research topic, with widespread applications. Most methods for visual recognition are fully supervised, as they make use of bounding boxes or pixelwise segmentations to locate objects of interest. However, this type of manual labeling is time consuming, error prone and it has been shown that manual segmentations are not necessarily the optimal spatial enclosure for object classifiers. This paper proposes a weakly-supervised system for multi-label image classification. In this setting, training images are annotated with a set of keywords describing their contents, but the visual concepts are not explicitly segmented in the images. We formulate the weakly-supervised image classification as a low-rank matrix completion problem. Compared to previous work, our proposed framework has three advantages: (1) Unlike existing solutions based on multiple-instance learning methods, our model is convex. We propose two alternative algorithms for matrix completion specifically tailored to visual data, and prove their convergence. (2) Unlike existing discriminative methods, our algorithm is robust to labeling errors, background noise and partial occlusions. (3) Our method can potentially be used for semantic segmentation. Experimental validation on several data sets shows that our method outperforms state-of-the-art classification algorithms, while effectively capturing each class appearance.

The current paper addresses the problem of object identification from multiple3D partial views, collected from different view angles with the objective of disambiguating between similar objects. We assume a mobile robot equipped with a depth sensor that autonomously collects observations from an object from different positions, with no previous known pattern. The challenge is to efficiently combine the set of observations into a single classification. We approach the problem with a multiple hypothesis filter that allows to combine information from a sequence of observations given the robot movement. We further innovate by off-line learning neighborhoods between possible hypothesis based on the similarity of observations. Such neighborhoods translate directly the ambiguity between objects, and allow to transfer the knowledge of one object to the other. In this paper we introduce our algorithm, Multiple Hypothesis for Object Class Disambiguation from Multiple Observations, and evaluate its accuracy and efficiency.

We propose a new model for simultaneously localizing different classes in the same media, casting it as an integer optimization problem. Our model subsumes into a single formulation previous single and multi-class localization methods, as well as allows us to exploit optimal relaxations to the linear domain. We apply our model to the problem of multi-label multiple instance learning for tagging video collections. Given weakly labeled training samples, where tags for actions in video and objects in images are known but not their locations, our aim is to train classifiers for both detection and localization of said classes on new data. Experimental results demonstrate our approach obtains similar performances when compared to fully supervised methods.

We introduce the Partial View Heat Kernel (PVHK) descriptor, for the purpose of 3D object representation and recognition from partial views, assumed to be partial object surfaces under self occlusion. PVHK describes partial views in a geometrically meaningful way, i.e., by establishing a unique relation between the shape of the view and the descriptor. PVHK is also stable with respect to sensor noise and therefore adequate for sensors, such as the current active 3D cameras. Furthermore, PVHK takes full advantage of the dual 3D/RGB nature of current sensors and seamlessly incorporates appearance information onto the 3D information. We formally define the PVHK descriptor, discuss related work, provide evidence of the PVHK properties and validate them in three purposefully diverse datasets, and demonstrate its potential for recognition tasks.

Low rank models have been widely used for the representation of shape, appearance or motion in computer vision problems. Traditional approaches to fit low rank models make use of an explicit bilinear factorization. These approaches benefit from fast numerical methods for optimization and easy kernelization. However, they suffer from serious local minima problems depending on the loss function and the amount/type of missing data. Recently, these low-rank models have alternatively been formulated as convex problems using the nuclear norm regularizer, unlike factorization methods, their numerical solvers are slow and it is unclear how to kernelize them or to impose a rank a priori. This paper proposes a unified approach to bilinear factorization and nuclear norm regularization, that inherits the benefits of both. We analyze the conditions under which these approaches are equivalent. Moreover, based on this analysis, we propose a new optimization algorithm and a "rank continuation'' strategy that outperform state-of-the-art approaches for Robust PCA, Structure from Motion and Photometric Stereo with outliers and missing data.

In this paper we propose a point matching algorithm that computes correspondences between images and/or 3D objects in affine camera settings. We formulate the point correspondence problem by minimizing an error function over the set of all binary decisions. This function has two components: a geometric error (akin to retro-projection error) and an image dissimilarity component. Originally a combinatorial problem we obtain its exact solution through a convex relaxation. Hinging on a recent theorem, this solution is derived by carefully designing the minimizing function. The large scale optimization problem is handled by an extremely fast algorithm based on Nesterov's projected gradient method. Experimental results show both computational efficiency and high robustness to large deviations and we demonstrate that it can cope with very hard real situations such as scenes with repetitive patterns. The methodology can be applied seamlessly to both 2D or/and 3D cameras.

We address the problem of incrementally recovering a matrix of tracked image points, based on partial observations of their trajectories. Besides partial observability, we assume the existence of gross, but sparse, noise on the known entries. This problem has obvious applications in real-time tracking and structure from motion, where observations are plagued by self-occlusion and outliers. Recently, work in the optimization community has spun optimal methods for matrix completion when this matrix is known to be low rank by minimizing the nuclear norm, the sum of its singular values. Despite exhibiting several optimality properties, no available algorithms perform this minimization incrementally. In this paper, we build upon the Nuclear Norm Robust PCA method and SPectrally Optimal Completion to propose a fast and incremental algorithm which is able to cope with outliers. We present experiments showing the competitive speed of our method while maintaining performance comparable to the state-of-the-art.

Recently, image categorization has been an active research topic due to the urgent need to retrieve and browse digital images via semantic keywords. This paper formulates image categorization as a multi-label classification problem using recent advances in matrix completion. Under this setting, classification of testing data is posed as a problem of completing unknown label entries on a data matrix that concatenates training and testing features with training labels. We propose two convex algorithms for matrix completion based on a Rank Minimization criterion specifically tailored to visual data, and prove its convergence properties. A major advantage of our approach w.r.t. standard discriminative classification methods for image categorization is its robustness to outliers, background noise and partial occlusions both in the feature and label space. Experimental validation on several datasets shows how our method outperforms state-of-the-art algorithms, while effectively capturing semantic concepts of classes.

This paper deals with the reconstruction of smooth, flexible, flat surfaces isometrically embedded in 3D when observed from uncalibrated camera images. It provides a cost function which captures the global geometry of the problem. The input data consists of a sequence of matched feature points (i.e. known correspondence), but the algorithm allows for missing data so that all points do not need to be visible in all images. Its performance under these conditions is analyzed. A way of inferring the bend radius at each point is provided which can be used back in the cost function, providing a higher order fitting. The performance is validated using both synthetic data and real image examples, where the reconstruction is compared to ground truth and reference points.

This paper deals with the reconstruction of smooth, flexible, isometrically embedded flat surfaces in 3D, such as a sheet of paper or a flag waving in the wind, from a set of 2D projected observations such as camera images. To solve the problem, a set of matched features of the waving object at different poses is needed, which are then applied to the reconstruction algorithm here described. The complete algorithm consists of 2 phases, the first obtaining an initial approximation from local features, the second uses this result to iterate a global cost function, trying to achieve a better estimate. To validate the algorithm, synthetic data with noise is generated, reconstructed and compared to ground truth data. Also, a second experiment consisting of real images of a sheet of paper is shown.

Finding correspondences between feature points is one of the most relevant problems in the whole set of visual tasks. In this paper we address the problem of matching a feature vector (or a matrix) to a given subspace. Given any vector base of such a subspace, we observe a linear combination of its elements with all entries swapped by an unknown permutation. We prove that such a computationally hard integer problem is uniquely solved in a convex set resulting from relaxing the original problem. Also, if noise is present, based on this result, we provide a robust estimate recurring to a linear programming-based algorithm. We use structure-from-motion and object recognition as motivating examples.

Perceiving dynamic scenes of rigid bodies, through affine projections of moving 3D point clouds, boils down to clustering the rigid motion subspaces supported by the points' image trajectories. For a physically meaningful interpretation, clusters must be consistent with the geometry of the underlying subspaces. Most of the existing measures for subspace clustering are ambiguous, or geometrically inconsistent. A practical consequence is that methods based on such (dis)similarities are unstable when the number of rigid bodies increase. This paper introduces the Normalized Subspace Inclusion (NSI) criterion to resolve these issues. Relying on this similarity, we propose a robust methodology for rigid motion segmentation, and test it, extensively, on the Hopkins155 database. The geometric consistency of the NSI assures the method's accuracy when the number of rigid bodies increases, while robustness proves to be suitable for dealing with challenging imaging conditions.

The 3D object recognition from a single or multiple 2D images is a very important problem in the computer vision field with a wide range of real applications. Considering the affine camera model, the main issue in solving this problem is the matching process between the object's 3D points and their 2D projections. In this work, we tackle the 3D-2D matching problem. It is formulated as a finite set of independent linear programs, solved efficiently. The 2D-2D and 3D-3D are also discussed. To show the validity of the proposed method, synthetic and real experiments are performed.

Segmenting arbitrary unions of linear subspaces is an important tool for computer vision tasks such as motion and image segmentation, SfM or object recognition. We segment subspaces by searching for the orthogonal complement of the subspace supported by the majority of the observations, i.e., the maximum consensus subspace. It is formulated as a Grassmannian optimization problem: a smooth, constrained but nonconvex program is immersed into the Grassmann manifold, resulting in a low dimensional and unconstrained program solved with an efficient optimization algorithm. Nonconvexity implies that global optimality depends on the initialization. However, by finding the maximum consensus subspace, outlier rejection becomes an inherent property of the method. Besides robustness, it does not rely on prior global detection procedures (e.g., rank of data matrices), which is the case of most current works. We test our algorithm in both synthetic and real data, where no outlier was ever classified as inlier.

In this article we explore the use of methodologies for 3D reconstruction from multiple images to recognize faces. We try to devise a strategy to tackle the problem of recognizing faces from images exhibiting strong pose (rotation and occlusion) and without prior knowledge (uncalibrated cameras, images from different sources). We do so by framing the recognition in the context of 3D structure from motion with missing data problems. In fact, recently, there has been a strong trend towards using 3D information to verify and recognize faces. However most of the state of the art works are developed over 3D sensors (3D range finders, stereo).Here we propose to do recognition by measuring the likelihood that one or more images were generated by a given 3D shape. In other words, given the subject's 3D shape, we compute the required transformation to match the probe images and measure its deviation from rigidity. This process is not straightforward due to the strong pose: face points are not visible in all images so there is a ldquomissing datardquo problem intrinsic to the formulation. One key assumption is that all images (and shape) correspond to neutral expression. Though limited in scale due to lack of large databases, a set of tests demonstrate the adequacy and good performance of the approach.

We address the problem of computing the Riemannian centroid of a constellation of points in a naturally reductive homogeneous manifold. We note that many interesting manifolds used in engineering (such as the special orthogonal group, Grassman, sphere, positive definite matrices) possess this structure. We develop an intrinsic Newton scheme for the centroid computation. This is achieved by exploiting a formula that we introduce for obtaining the Hessian of the squared Riemannian distance on naturally reductive homogeneous spaces. Some results of finding the centroid of a constellation of points in these spaces are presented, which evidence the quadratic convergence of the Newton method derived herein. These computer simulation results show that, as expected, the Newton method has a faster convergence rate than the usual gradient-based approaches

We propose a solution to the n-frame correspondence problem under the factorization framework. During the matching process, our algorithm takes explicitly into account the geometrical constraints associated to the reconstruction process. To this end, a rank constraint is imposed on the measurement matrix. Since our method relies solely on geometric constraints, it is not dependent on corners as image features and can consequently match generic points (e.g. contours). Outlier rejection is integrated as a part of the actual matching process. In general the problem is formulated as combinatorial, but we develop a method, which can provide a solution with a low computational complexity. Because of this, our algorithm is able to handle high-dimensional matching problems that are common in real-life applications.

We propose a new methodology for reliably solving the correspondence problem between sparse sets of points of two or more images. This is a key step inmost problems of computer vision and, so far, no general method exists to solve it. Our methodology is able to handle most of the commonly used assumptions in a unique formulation, independent of the domain of application and type of features. It performs correspondence and outlier rejection in a single step and achieves global optimality with feasible computation. Feature selection and correspondence are first formulated as an integer optimization problem. This is a blunt formulation, which considers the whole combinatorial space of possible point selections and correspondences. To find its global optimal solution, we build a concave objective function and relax the search domain into its convex-hull. The special structure of this extended problem assures its equivalence to the original one, but it can be optimally solved by efficient algorithms that avoid combinatorial search. This methodology can use any criterion provided it can be translated into cost functions with continuous second derivatives.

We propose a new methodology for reliably solving the correspondence problem between points of two or more images. This is a key step in most problems of Computer Vision and, so far, no general method exists to solve it. Our methodology is able to handle most of the commonly used assumptions in a unique formulation, independent of the domain of application and type of features. It performs correspondence and outlier rejection in a single step, and achieves global optimality with feasible computation. Feature selection and correspondence are first formulated as an integer optimization problem. This is a blunt formulation, which considers the whole combinatorial space of possible point selections and correspondences. To find its global optimal solution we build a concave objective function and relax the search domain into its convex-hull. The special structure of this extended problem assures its equivalence to the original one, but it can be optimally solved by efficient algorithms that avoid combinatorial search.

This paper presents a method to automatically synthesize human face images from holistic descriptions. We compactly represent the face set by a small set of prototypes, which can be used in simple ways to generate controlled morphings. This becomes possible because separation of 2D-shape and texture provides a faithful, closed and convex representation of images, and smoothes the mappings between images and their properties. With this approach, the user watches an image being continuously morphed according to his indications, and the synthesized images always obey the natural physiognomic constraints.

The structure from motion problem has been extensively studied in the field of computer vision. Yet, the bulk of the existing work assumes that the scene contains only a single moving object. The more realistic case where an unknown number of objects move in the scene has received little attention, especially for its theoretical treatment. We present a new method for separating and recovering the motion and shape of multiple independently moving objects in a sequence of images. The method does not require prior knowledge of the number of objects, nor is dependent on any grouping of features into an object at the image level. For this purpose, we introduce a mathematical construct of object shapes, called the shape interaction matrix, which is invariant to both the object motions and the selection of coordinate systems. This invariant structure is computable solely from the observed trajectories of image features without grouping them into individual objects. Once the structure is computed, it allows for segmenting features into objects by the process of transforming it into a canonical form, as well as recovering the shape and motion of each object.>