This paper addresses the challenge of devising new representation learning algorithms that overcome the lack of interpretability of classical visual models. Therefore, it introduces a new recursive visual patch selection technique built on top of a Shared Nearest Neighbors embedding method. The main contribution of the paper is to drastically reduce the high-dimensionality of such over-complete representation thanks to a recursive feature elimination method. We show that the number of spatial atoms of the representation can be reduced by up to two orders of magnitude without much degrading the encoded information. The resulting representations are shown to provide competitive image classification performance with the state-of-the-art while enabling to learn highly interpretable visual models.

This paper presents an automatic system able to identify visual named-entities appearing in images and videos, among a list of 25,000 entities, aggregated from Wikipedia lists, and more specific websites. DigInPix is a generic application designed to identify different kinds of entities. In this first attempt, we only focus on logo identification (more generally on legal persons). The identification process mainly relies on an efficient CBIR system, searching in an indexed image database composed of 600,000 weak-labelled images crawled from Google Images. DigInPix proposes a responsive-design html5 interface [1] for testing purposes.

This paper introduces a new image representation relying on the spatial pooling of geometrically consistent visual matches. We therefore introduce a new match kernel based on the inverse rank of the shared nearest neighbors combined with local geometric constraints. To avoid overfitting and reduce processing costs, the dimensionality of the resulting over-complete representation is further reduced by hierarchically pooling the raw consistent matches according to their spatial position in the training images. The final image representation is obtained by concatenating the resulting feature vectors at several resolutions. Learning from these representations using a logistic regression classifier is shown to provide excellent fine-grained classification performances outperforming the results reported in the literature on several classification tasks.

This paper considers the problem of recognizing legal entities in visual contents in a similar way to named-entity recognizers for text documents. Whereas previous works were restricted to the recognition of a few tens of logotypes, we generalize the problem to the recognition of thousands of legal persons, each being modeled by a rich corporate identity automatically built from web images. We introduce a new geometrically-consistent instance-based classification method that is shown to outperform state-of-the-art techniques on several challenging datasets while being much more scalable. Further experiments performed on an automatic web crawl of 5,824 legal entities demonstrates the scalability of the approach.

Who said What, Where and How? How are images, video and stories spreading out? Who produces the information? OTMedia addresses these questions by collecting, enriching and analysing continuously more than 1500 streams of French media from TV Radio, Web, AFP, and Twitter. Two studies on media produced by end users with the OTMedia framework are presented.

State-of-the-art visual search methods allow retrieving efficiently small rigid objects in very large image datasets (e.g. logos, paintings, etc.). User's perception of the classical query-by-window paradigm is however affected by the fact that many submitted queries actually return nothing or only junk results. We demonstrate in this demo that the perception can be radically different if the objects of interest are rather suggested to the user by pre-computing relevant clusters of instances. Impressive results involving very small objects discovered in a web collection of 110K images are demonstrated through a simple interactive GUI.

This paper presents a visual-based media event detection system based on the automatic discovery of the most circulated images across the main news media (news websites, press agencies, TV news and newspapers). Its main originality is to rely on the transmedia contextual information to denoise the raw visual detections and consequently focus on the most salient transmedia events.

This paper presents a scalable method for automatically discovering frequent visual objects in large multimedia collections even if their size is very small. It first formally revisits the problem of mining or discovering such objects, and then generalizes two kinds of existing methods for probing candidate object seeds: weighted adaptive sampling and hashing-based methods. The idea is that the collision frequencies obtained with hashing-based methods can actually be converted into a prior probability density function given as input to a weighted adaptive sampling algorithm. This allows for an evaluation of any hashing scheme effectiveness in a more generalized way, and a comparison with other priors, e.g. guided by visual saliency concerns. We then introduce a new hashing strategy, working first at the visual level, and then at the geometric level. This strategy allows us to integrate weak geometric constraints into the hashing phase itself and not only neighborhood constraints as in previous works. Experiments conducted on a new dataset introduced in this paper will show that using this new hashing-based prior allows a drastic reduction of the number of tentative probes required to discover small objects instantiated several times in a large dataset.

Following the success of hashing methods for multidimensional indexing, more and more works are interested in embedding visual feature space in compact hash codes. Such approaches are not an alternative to using index structures but a complementary way to reduce both the memory usage and the distance computation cost. Several data dependent hash functions have notably been proposed to closely fit data distribution and provide better selectivity than usual random projections such as LSH. However, improvements occur only for relatively small hash code sizes up to 64 or 128 bits. As discussed in the paper, this is mainly due to the lack of independence between the produced hash functions. We introduce a new hash function family that attempts to solve this issue in any kernel space. Rather than boosting the collision probability of close points, our method focus on data scattering. By training purely random splits of the data, regardless the closeness of the training samples, it is indeed possible to generate consistently more independent hash functions. On the other side, the use of large margin classifiers allows to maintain good generalization performances. Experiments show that our new Random Maximum Margin Hashing scheme (RMMH) outperforms four state-of-the-art hashing methods, notably in kernel spaces.

State-of-the-art large-scale object retrieval systems usually combine efficient Bag-of-Words indexing models with a spatial verification re-ranking stage to improve query performance. In this paper we propose to directly discover spatially verified visual words as a batch process. Contrary to previous related methods based on feature sets hashing or clustering, we suggest not trading recall for efficiency by sticking on an accurate two-stage matching strategy. The problem then rather becomes a sampling issue: how to effectively and efficiently select relevant query regions while minimizing the number of tentative probes? We therefore introduce an adaptive weighted sampling scheme, starting with some prior distribution and iteratively converging to unvisited regions. Interestingly, the proposed paradigm is generalizable to any input prior distribution, including specific visual concept detectors or efficient hashing-based methods. We show in the experiments that the proposed method allows to discover highly interpretable visual words while providing excellent recall and image representativity.

This paper presents 3 interfaces to access video contents. The stream explorer allows to explore and to segment video streams. The video explorer shows a synthetic view of structured TV programmes. The collection explorer proposes cartographies of large video collections. Based on visual and textual automatic processing, proximities and redundancies are analyzed, allowing the emergence of different levels of structure. This is made possible thanks to the volume of data considered: 7 channels during 100 days, ie 16000 hours or 20 Millions key frames. These three tools allow efficient exploration of video contents at different levels of interest: image, shot and sequence, programme and collection level.

This paper presents a new content-based retrieval framework applied to logo retrieval in large natural image collections. The first contribution is a new challenging dataset, called BelgaLogos, which was created in collaboration with professionals of a press agency, in order to evaluate logo retrieval technologies in real-world scenarios. The second and main contribution is a new visual query expansion method using an a contrario thresholding strategy in order to improve the accuracy of expanded query images. Whereas previous methods based on the same paradigm used a purely hand tuned fixed threshold, we provide a fully adaptive method enhancing both genericity and effectiveness. This new technique is evaluated on both OxfordBuilding dataset and our new BelgaLogos dataset.

Efficient high-dimensional similarity search structures are essential for building scalable content-based search systems on feature-rich multimedia data. In the last decade, Locality Sensitive Hashing (LSH) has been proposed as indexing technique for approximate similarity search. Among the most recent variations of LSH, multi-probe LSH techniques have been proved to overcome the overlinear space cost drawback of common LSH. Multi-probe LSH is built on the well-known LSH technique, but it intelligently probes multiple buckets that are likely to contain query results in a hash table. Our method is inspired by previous work on probabilistic similarity search structures and improves upon recent theoretical work on multi-probe and query adaptive LSH. Whereas these methods are based on likelihood criteria that a given bucket contains query results, we define a more reliable a posteriori model taking account some prior about the queries and the searched objects. This prior knowledge allows a better quality control of the search and a more accurate selection of the most probable buckets. We implemented a nearest neighbors search based on this paradigm and performed experiments on different real visual features datasets. We show that our a posteriori scheme outperforms other multi-probe LSH while offering a better quality control. Comparisons to the basic LSH technique show that our method allows consistent improvements both in space and time efficiency.

Mining the video content itself can bring to light important information regarding the internal structure of large video databases, compensating for a lasting absence of extensive and reliable annotations. Many valuable links between video segments can be identified by content-based copy detection methods, where copies are transformed versions of original video sequences. To make this approach viable for large video databases, we put forward a new mining method relying on the definition of a compact keyframe-level descriptor and of a specific index structure. The performance obtained in detecting links between video segments is evaluated with the help of a ground truth and several illustrations are given. The scalability of the approach is then demonstrated for databases of up to 10,000 hours of video.

This paper presents applications for dealing with videos on the Web, using an efficient technique for video copy detection in large archives. Managing videos on the Web is the source of two exciting challenges: the respect of the copyright and the linkage of multiple videos. We present a technique called ViCopT for video copy tracking which is based on labels of behavior of local descriptors computed along video. The results obtained on large amount of data (270 hours of videos from the Internet) are very promising, even with a large video database (700 hours): ViCopT displays excellent robustness to various severe signal transformations, making it able to identify copies accurately from highly similar videos, as well as to link similar videos, in order to reduce redundancy or to gather the metadata associated. Finally, we also show that ViCopT goes further by detecting segments having the same background, with the aim of linking videos of the same category, like forecast weather programs or particular TV shows.

Content-based copy retrieval (CBCR) aims at retrieving in a database all the modified versions or the previous versions of a given candidate object. In this paper, we present a copy-retrieval scheme based on local features that can deal with very large databases both in terms of quality and speed. We first propose a new approximate similarity search technique in which the probabilistic selection of the feature space regions is not based on the distribution in the database but on the distribution of the features distortion. Since our CBCR framework is based on local features, the approximation can be strong and reduce drastically the amount of data to explore. Furthermore, we show how the discrimination of the global retrieval can be enhanced during its post-processing step, by considering only the geometrically consistent matches. This framework is applied to robust video copy retrieval and extensive experiments are presented to study the interactions between the approximate search and the retrieval efficiency. Largest used database contains more than 1 billion local features corresponding to 30000 h of video

This paper presents an efficient approach for copies detection in a large videos archive consisting of several hundred of hours. The video content indexing method consists of extracting the dynamic behavior on the local description of interest points and further on the estimation of their trajectories along the video sequence. Analyzing the low-level description obtained allows to highlight trends of behaviors and then to assign a label of behavior to each local descriptor. Such an indexing approach has several interesting properties: it provides a rich, compact and generic description, while labels of behavior provide a high-level description of the video content. Here, we focus on video Content Based Copy Detection (CBCD). Copy detection is problematic as similarity search problem but with prominent differences. To be efficient, it requires a dedicated on-line retrieval method based on a specific voting function. This voting function must be robust to signal transformations and discriminating versus high similarities which are not copies. The method we propose in this paper is a dedicated on-line retrieval method based on a combination of the different dynamic contexts computed during the off-line indexing. A spatio-temporal registration based on the relevant combination of detected labels is then applied. This approach is evaluated using a huge video database of 300 hours with different video tests. The method is compared to a state-of-the art technique in the same conditions. We illustrate that taking labels into account in the specific voting process reduces false alarms significantly and drastically improves the precision.

Recent methods based on interest points and local fingerprints have been proposed to perform robust CBCD (content-based copy detection) of images and video. They include two steps: the search for similar local fingerprints in the database (DB) and a voting strategy that merges all the local results in order to perform a global decision. In most image or video retrieval systems, the search for similar features in the DB is performed by a geometrical query in a multidimensional index structure. Recently, the paradigm of approximate k-nearest neighbors query has shown that trading quality for time can be widely profitable in that context. In this paper, we evaluate a new approximate search paradigm, called statistical similarity search (S/sup 3/) in a complete CBCD scheme based on video local fingerprints. Experimental results show that these statistical queries allow high performance gains compared to classical e-range queries and that trading quality for time during the search does not degrade seriously the global robustness of the system, even with very large DBs including more than 20,000 hours of video.

In many image or video retrieval systems, the search for similar objects in the database includes a spatial access method to a multidimensional feature space. This step is generally considered as a problem independent of the features and the similarity type. The well known multidimensional nearest neighbor search has also been widely studied by the database community as a generic method. We propose a novel strategy dedicated to pseudo-invariant features retrieval and more specifically applied to content based copy identification. The range of a query is computed during the search according to deviation statistics between original and observed features. Furthermore, this approximate search range is directly mapped onto a Hilbert space-filling curve, allowing an efficient access to the database. Experimental results give excellent response times for very large databases both on synthetic and real data. This work is used in a TV monitoring system including more than 13000 hours of video in the reference database.

Line scratches are common degradations in motion picture films. This paper presents an efficient method for line scratches detection strengthened by a Kalman filter. A new interpolation technique, dealing with both low and high frequencies (i.e. film grain) around the line artifacts, is investigated to achieve a nearby invisible reconstruction of damaged areas. Our line scratches detection and removal techniques have been validated on several film sequences.

This paper presents a robust technique to detect local deteriorations of old cinematographic films. This method relies on spatio-temporal information and combines two different detectors: a morphological detector which uses spatial properties of deteriorations, and a dynamic detector based on motion estimation techniques. Our deterioration detector has been validated on several film sequences and turned out to be a powerful tool for digital film restoration.