We present a novel face reconstruction method capable of reconstructing detailed face geometry, spatially varying face reflectance from a single monocular image. We build our work upon the recent advances of DNN-based auto-encoders with differentiable ray tracing image formation, trained in self-supervised manner. While providing the advantage of learning-based approaches and real-time reconstruction, the latter methods lacked fidelity. In this work, we achieve, for the first time, high fidelity face reconstruction using self-supervised learning only. Our novel coarse-to-fine deep architecture allows us to solve the challenging problem of decoupling face reflectance from geometry using a single image, at high computational speed. Compared to state-of-the-art methods, our method achieves more visually appealing reconstruction.

Robust face reconstruction from monocular image in general lighting conditions is challenging. Methods combining deep neural network encoders with differentiable rendering have opened up the path for very fast monocular reconstruction of geometry, lighting and reflectance. They can also be trained in self-supervised manner for increased robustness and better generalization. However, their differentiable rasterization-based image formation models, as well as underlying scene parameterization, limit them to Lambertian face reflectance and to poor shape details. More recently, ray tracing was introduced for monocular face reconstruction within a classic optimization-based framework and enables state-of-the art results. However, optimization-based approaches are inherently slow and lack robustness. In this paper, we build our work on the afore-mentioned approaches and propose a new method that greatly improves reconstruction quality and robustness in general scenes. We achieve this by combining a CNN encoder with a differentiable ray tracer, which enables us to base the reconstruction on much more advanced personalized diffuse and specular albedos, a more sophisticated illumination model and a plausible representation of self-shadows. This enables to take a big leap forward in reconstruction quality of shape, appearance and lighting even in scenes with difficult illumination. With consistent face attributes reconstruction, our method leads to practical applications such as relighting and self-shadows removal. Compared to state-of-the-art methods, our results show improved accuracy and validity of the approach.

Abstract We present a differentiable ray‐tracing based novel face reconstruction approach where scene attributes – 3D geometry, reflectance (diffuse, specular and roughness), pose, camera parameters, and scene illumination – are estimated from unconstrained monocular images. The proposed method models scene illumination via a novel, parameterized virtual light stage, which in‐conjunction with differentiable ray‐tracing, introduces a coarse‐to‐fine optimization formulation for face reconstruction. Our method can not only handle unconstrained illumination and self‐shadows conditions, but also estimates diffuse and specular albedos. To estimate the face attributes consistently and with practical semantics, a two‐stage optimization strategy systematically uses a subset of parametric attributes, where subsequent attribute estimations factor those previously estimated. For example, self‐shadows estimated during the first stage, later prevent its baking into the personalized diffuse and specular albedos in the second stage. We show the efficacy of our approach in several real‐world scenarios, where face attributes can be estimated even under extreme illumination conditions. Ablation studies, analyses and comparisons against several recent state‐of‐the‐art methods show improved accuracy and versatility of our approach. With consistent face attributes reconstruction, our method leads to several style – illumination, albedo, self‐shadow – edit and transfer applications, as discussed in the paper.

In object recognition, the Bag-of-Words model assumes: i) extraction of local descriptors from images, ii) embedding the descriptors by a coder to a given visual vocabulary space which results in mid-level features, iii) extracting statistics from mid-level features with a pooling operator that aggregates occurrences of visual words in images into signatures, which we refer to as First-order Occurrence Pooling. This paper investigates higher-order pooling that aggregates over co-occurrences of visual words. We derive Bag-of-Words with Higher-order Occurrence Pooling based on linearisation of Minor Polynomial Kernel, and extend this model to work with various pooling operators. This approach is then effectively used for fusion of various descriptor types. Moreover, we introduce Higher-order Occurrence Pooling performed directly on local image descriptors as well as a novel pooling operator that reduces the correlation in the image signatures. Finally, First-, Second-, and Third-order Occurrence Pooling are evaluated given various coders and pooling operators on several widely used benchmarks. The proposed methods are compared to other approaches such as Fisher Vector Encoding and demonstrate improved results.

Regarding texture features, Local-based methods such as Local Binary Pattern (LBP) and its variants are computationally efficient high-performing but sensitive to noise, and suffering global structure information loss. By contrast, filter-based counterparts, the Scattering Transform for instance, are tolerant to noise and translation but often lack of small local structure information. In this paper we propose an integration of those to take full advantages of both local and global features. In this way, LBP is used for extracting local features while the Scattering Transform feature plays the role of a global descriptor. In addition to the combination of these two state-of-the-art features, we further integrate a new preprocessing technique called biologically-inspired filtering (BF) as well as an efficient PCA classifier. Intensive experiments conducted on many texture benchmarks such as CUReT, UIUC, KTH-TIPS2b, and OUTEX show that our combined method not only outweighs each one which stands alone but also competes with state-of-the-art on the experimented datasets.

We consider a pipeline for image classification or search based on coding approaches like bag of words or Fisher vectors. In this context, the most common approach is to extract the image patches regularly in a dense manner on several scales. This paper proposes and evaluates alternative choices to extract patches densely. Beyond simple strategies derived from regular interest region detectors, we propose approaches based on superpixels, edges, and a bank of Zernike filters used as detectors. The different approaches are evaluated on recent image retrieval and fine-grained classification benchmarks. Our results show that the regular dense detector is outperformed by other methods in most situations, leading us to improve the state-of-the-art in comparable setups on standard retrieval and fined-grained benchmarks. As a byproduct of our study, we show that existing methods for blob and superpixel extraction achieve high accuracy if the patches are extracted along the edges and not around the detected regions.

Several descriptors have been proposed in the past for 3D shape analysis, yet none of them achieves best performance on all shape classes. In this paper we propose a novel method for 3D shape analysis using the covariance matrices of the descriptors rather than the descriptors themselves. Covariance matrices enable efficient fusion of different types of features and modalities. They capture, using the same representation, not only the geometric and the spatial properties of a shape region but also the correlation of these properties within the region. Covariance matrices, however, lie on the manifold of Symmetric Positive Definite (SPD) tensors, a special type of Riemannian manifolds, which makes comparison and clustering of such matrices challenging. In this paper we study covariance matrices in their native space and make use of geodesic distances on the manifold as a dissimilarity measure. We demonstrate the performance of this metric on 3D face matching and recognition tasks. We then generalize the Bag of Features paradigm, originally designed in Euclidean spaces, to the Riemannian manifold of SPD matrices. We propose a new clustering procedure that takes into account the geometry of the Riemannian manifold. We evaluate the performance of the proposed Bag of Covariance Matrices framework on 3D shape matching and retrieval applications and demonstrate its superiority compared to descriptor-based techniques.

In web-scale image retrieval, the most effective strategy is to aggregate local descriptors into a high dimensionality signature and then reduce it to a small dimensionality. Thanks to this strategy, web-scale image databases can be represented with small index and explored using fast visual similarities. However, the computation of this index has a very high complexity, because of the high dimensionality of signature projectors. In this work, we propose a new efficient method to greatly reduce the signature dimensionality with low computational and storage costs. Our method is based on the linear projection of the signature onto a small subspace using a sparse projection matrix. We report several experimental results on two standard datasets (Inria Holidays and Oxford) and with 100k image distractors. We show that our method reduces both the projectors storage cost and the computational cost of projection step while incurring a very slight loss in mAP (mean Average Precision) performance of these computed signatures.

In this paper, we tackle the storage and computational cost of linear projections used in dimensionality reduction for near duplicate image retrieval. We propose a new method based on metric learning with a lower training cost than existing methods. Moreover, by adding a sparsity constraint, we obtain a projection matrix with a low storage and projection cost. We carry out experiments on a well known near duplicate image dataset and show our algorithm behaves correctly. Retrieval performances are shown to be promising when compared to the memory footprint and the projection cost of the obtained sparse matrix.

Elastic shape analysis on non-linear Riemannian manifolds provides an efficient and elegant way for simultaneous comparison and registration of non-rigid shapes. In such formulation, shapes become points on some high dimensional shape space. A geodesic between two points corresponds to the optimal deformation needed to register one shape onto another. The length of the geodesic provides a proper metric for shape comparison. However, the computation of geodesics, and therefore the metric, is computationally very expensive as it involves a search over the space of all possible rotations and re- parameterization. This problem is even more important in shape retrieval scenarios where the query shape is compared to every element in the collection to search. In this paper, we propose a new procedure for metric approximation using the framework of kernel functions. We will demonstrate that this provides a fast approximation of the metric while preserving its invariance properties.

This paper addresses the problem of large scale image representation for object recognition and classification. Our work deals with the problem of optimizing the classification accuracy and the dimensionality of the image representation. We propose to iteratively select sets of projections from an external dataset, using Bagging and feature selection thanks to SVM normals. Features are selected using weights of SVM normals in orthogonalized sets of projections. The Bagging strategy is employed to improve the results and provide more stable selection. The overall algorithm linearly scales with the size of features, and thus is able to process the large state-of-the-art image representation. Given Spatial Fisher Vectors as input, our method consistently improves the classification accuracy for smaller vector dimensionality, as demonstrated by our results on the popular and challenging PASCAL VOC 2007 benchmark.

We propose a new method for approximate k-NN search in large scale image databases, based on top-k multi-criteria search techniques. The method defines a simple index structure based on sorted lists, which provides a good compromise between fast retrieval, storage requirements and update cost. The search algorithm delivers approximate results with guarantees about false negatives, with fast emergence of good approximations, monotonically improved and leading if necessary to an exact result. Experiments with the on-disk implementation show that our method produces very good approximate results several times faster than the Baseline method.

In this paper, we propose a unified formalism for video descriptors. This formalism is based on the descriptors decomposition in three levels: primitive, scattering and projection. With this framework, we are able to rewrite easily all the usual descriptors in the literature such as HOG, HOF, SURF. Then, we propose a new projection method based on approximation with a finite expansion of orthogonal polynomials. Using our framework, we extend all usual descriptors by switching the projection step. The experiments are carried out on the well known KTH dataset and on the more challenging Hollywood 2 action classification dataset and show state of the art results.

In this paper, we present an efficient 3D object retrieval method invariant to scale, orientation and pose. Our approach is based on the dense extraction of discriminative local descriptors extracted from 2D views. We aggregate the descriptors into a single vector signature using tensor products. The similarity between 3D models can then be efficiently computed with a simple dot product. Experiments on the SHREC12 commonly-used benchmark demonstrate that our approach obtains superior performance in searching for generic shapes.

Many methods have been recently proposed to deal with the large amount of data provided by high-resolution remote sensing technologies. Several of these methods rely on the use of image segmentation algorithms for delineating target objects. However, a common issue in geographic object-based applications is the definition of the appropriate data representation scale, a problem that can be addressed by exploiting multiscale segmentation. The use of multiple scales, however, raises new challenges related to the definition of effective and efficient mechanisms for extracting features. In this paper, we address the problem of extracting histogram-based features from a hierarchy of regions for multiscale classification. The strategy, called H-Propagation, exploits the existing relationships among regions in a hierarchy to iteratively propagate features along multiple scales. The proposed method speeds up the feature extraction process and yields good results when compared with global low-level extraction approaches.

Recent works on image retrieval have proposed to index images by compact representations encoding powerful local descriptors, such as the closely related VLAD and Fisher vector. By combining such a representation with a suitable coding technique, it is possible to encode an image in a few dozen bytes while achieving excellent retrieval results. This paper revisits some assumptions proposed in this context regarding the handling of "visual burstiness", and shows that ad-hoc choices are implicitly done which are not desirable. Focusing on VLAD without loss of generality, we propose to modify several steps of the original design. Albeit simple, these modifications significantly improve VLAD and make it compare favorably against the state of the art.

In this paper, we propose a system for interactive image retrieval in dynamic databases, where images are regularly added or removed. In order to handle this, we propose a method that tunes itself according to user labels. The framework we propose is based on visual dictionaries, with the specificity that the dictionaries are built online, during retrieval sessions. In other words, each user has its own visual dictionary, as opposed to usual approaches where all users share the same visual dictionary. In order to create theses dictionaries, we propose a method based on kernel functions. This method iteratively selects base kernels from a large base kernel pool, where each base kernel is related to a low-level descriptor such as color or texture. This learning process is performed in real time, and the classification of the database is faster than usual techniques since only relevant features for the current query are used. Experiments are carried out on a generalist database, and show the ability of the method to build effective kernels with few labels.

In this paper, we propose a novel algorithm to design multi-class kernel functions based on an iterative combination of weak kernels in a scheme inspired from boosting framework. The method proposed in this article aims at building a new feature where the centroid for each class are optimally located. We evaluate our method for image categorization by considering a state-of-the-art image database and by comparing our results with reference methods. We show that on the Oxford Flower databases our approach achieves better results than previous state-of-the-art methods.

Within the Content Based Image Retrieval (CBIR) framework, one of the main challenges is to tackle the scalability issues. We propose a new compact signature for similarity search. We use an original method to perform a high compression of signatures while retraining their effectiveness. We propose an embedding method that maps large signatures into a low-dimensional Hilbert space. We evaluated the method on Holidays database and compared the results with methods of state-of-the-art.

A huge effort has been applied in image classification to create high-quality thematic maps and to establish precise inventories about land cover use. The peculiarities of remote sensing images (RSIs) combined with the traditional image classification challenges made RSI classification a hard task. Our aim is to propose a kind of boost-classifier adapted to multiscale segmentation. We use the paradigm of boosting, whose principle is to combine weak classifiers to build an efficient global one. Each weak classifier is trained for one level of the segmentation and one region descriptor. We have proposed and tested weak classifiers based on linear support vector machines (SVM) and region distances provided by descriptors. The experiments were performed on a large image of coffee plantations. We have shown in this paper that our approach based on boosting can detect the scale and set of features best suited to a particular training set. We have also shown that hierarchical multiscale analysis is able to reduce training time and to produce a stronger classifier. We compare the proposed methods with a baseline based on SVM with radial basis function kernel. The results show that the proposed methods outperform the baseline.

Within the Content Based Image Retrieval (CBIR) framework, three main points can be highlighted: visual descriptors extraction, image signatures and their associated similarity measures, and machine learning based relevance functions. While the first and the last points have vastly improved in re- cent years, this paper addresses the second point. We propose a novel approach to compute vector representations extending state of the art methods in the field. Furthermore, our method can be viewed as a linearization of efficient well known kernel methods. The evaluation shows that our representation significantly improve state of the art results on the difficult VOC2007 database by a fair margin.

Recent approaches of graph comparison consider graphs as sets of paths. Kernels on graphs are then computed from kernels on paths. A common strategy for graph retrieval is to perform pairwise comparisons. In this paper, we propose to follow a different strategy, where we collect a set of paths into a dictionary, and then project each graph to this dictionary. Then, graphs can be classified using powerful classification methods, such as SVM. Furthermore, we collect the paths through interaction with a user. This strategy is ten times faster than a straight comparisons of paths. Experiments have been carried out on a database of city windows.

This paper presents a new algorithm based on boosting for interactive object retrieval in images. Recent works propose ”online boosting” algorithms where weak classifier sets are iteratively trained from data. These algorithms are proposed for visual tracking in videos, and are not well adapted to ”online boosting” for interactive retrieval. We propose in this paper to iteratively build weak classifiers from images, labeled as positive by the user during a retrieval session. A novel active learning strategy for the selection of images for user annotation is also proposed. This strategy is used to enhance the strong classifier resulting from ”boosting” process, but also to build new weak classifiers. Experiments have been carried out on a generalist database in order to compare the proposed method to a SVM based reference approach.

In the framework of the interactive search in image databases, we are interested in similarity measures able to learn during the search and usable in real-time. Images are represented by adjacency graphs of regions. In order to compare attributed graphs, we employ kernels on graphs built on sets of paths. In this paper, we introduce a fast kernel function whose similarity is based on several matches. We also introduce new features for edges in the graph. Experiments on a specific database having objects with heterogeneous backgrounds show the performance of our object retrieval technique.

Active learning methods have been considered with increased interest in the statistical learning community. Initially developed within a classification framework, a lot of extensions are now being proposed to handle multimedia applications. This paper provides algorithms within a statistical framework to extend active learning for online content-based image retrieval (CBIR). The classification framework is presented with experiments to compare several powerful classification techniques in this information retrieval context. Focusing on interactive methods, active learning strategy is then described. The limitations of this approach for CBIR are emphasized before presenting our new active selection process RETIN. First, as any active method is sensitive to the boundary estimation between classes, the RETIN strategy carries out a boundary correction to make the retrieval process more robust. Second, the criterion of generalization error to optimize the active learning selection is modified to better represent the CBIR objective of database ranking. Third, a batch processing of images is proposed. Our strategy leads to a fast and efficient active learning scheme to retrieve sets of online images (query concept). Experiments on large databases show that the RETIN method performs well in comparison to several other active strategies.

We propose in this paper a general kernel framework to deal with database object retrieval embedded in images with heterogeneous background. We use local features computed on fuzzy regions for image representation summarized in bags, and we propose original kernel functions to deal with sets of features and spatial constraints. Combined with SVMs classification and online learning scheme, the resulting algorithm satisfies the robustness requirements for representation and classification of objects. Experiments on a specific database having objects with heterogeneous backgrounds show the performance of our object retrieval technique.

The automatic computation of features for content-based image retrieval still has difficulties to represent the concepts the user has in mind. Whenever an additional learning strategy (such as relevance feedback) can improve the results of the search, the system performances still depend on the representation of the image collection. We introduce in this paper a supervised optimization of a set of feature vectors. According to an incomplete set of partial labels, the method improves the representation of the image collection, even if the size, the number, and the structure of the concepts are unknown. Experiments have been carried out on a large general database in order to validate our approach.

Active learning methods have been considered with an increased interest in the content-based image retrieval (CBIR) community. These methods have been developed for classification problems, and do not deal with the particular characteristics of the CBIR. One of these characteristics is the criterion to optimize, for instance the error of generalization for classification, which is not the best adapted to CBIR context. We introduce in this paper an active selection which chooses the image the user should label such as the mean average precision is increased. The method is smartly combined with previous propositions, and leads to a fast and efficient active learning scheme. Experiments on a large database have been carried out in order to compare our approach to several other methods.

We present a new system, called Retimm, for searching databases made of documents containing images and text. Images are indexed by color and texture distributions. Color and texture classes are obtained by a quantization adapted to the whole database. Signatures are ranked m times, once for each dimension, but values are not stored. The search engine works as a vote system: the score for each document is the total of the votes of all coordinates, these last votes depending on a k-nn search on each dimension. Retimm is able to retrieve very quickly images from large databases from any request composed of one or several images and/or one or several words. The system is interactive, since the query can be modified at any moment by adding or removing images or words

Content-based image retrieval systems still have difficulties to bridge the semantic gap between the low-level representation of images and the high level concepts the user is looking for. Relevance feedback methods deal with this problem using labels provided by users, but only during the current retrieval session. In this paper, we introduce a semantic learning method to manage user labels in CBIR applications. Our approach uses a kernel matrix to represent semantic information in a statistical learning framework. The kernel matrix is updated according to labels provided by users after retrieval sessions. Experiments have been carried out on a large generalist database in order to validate our approach.

Active learning methods have been considered with an increasing interest in the content-based image retrieval (CBIR) community. In this article, we propose an efficient method based on active learning strategy to retrieve large image categories. At each feedback step, the system optimizes the image set presented to the user in order to speed up the retrieval. Experimental tests on COREL photo database have been carried out.