Neural networks typically need huge amounts of data to train in order to get reasonable generalizable results. A common approach is to artificially generate samples by using prior knowledge of the data properties or other relevant domain knowledge. However, if the assumptions on the data properties are not accurate or the domain knowledge is irrelevant to the task at hand, one may end up degenerating learning performance by using such augmented data in comparison to simply training on the limited available dataset. We propose a critical data augmentation method using feature side-information, which is obtained from domain knowledge and provides detailed information about features' intrinsic properties. Most importantly, we introduce an instance wise quality checking procedure on the augmented data. It filters out irrelevant or harmful augmented data prior to entering the model. We validated this approach on both synthetic and real-world datasets, specifically in a scenario where the data augmentation is done based on a task independent, unreliable source of information. The experiments show that the introduced critical data augmentation scheme helps avoid performance degeneration resulting from incorporating wrong augmented data.

Indexing web-scale multimedia is only possible by distributing storage and computing efforts. Existing large-scale content-based indexing services mostly do not offer interactive relevance feedback. Here, we propose a running demonstrator of our Cross-Modal Search Engine (CMSE) implementing a query-by-example search strategy with relevance feedback and distributed over a cluster of 20 Dual core machines using MPI. We present the performance gain in terms of interactivity (search time) using a part of the Image-Net collection containing more than one million images as base example.

We demonstrate our TagCaptcha image annotation system. TagCaptcha presents the user with a number of images that must be correctly labelled in order to pass a human verification test on the web. The images are divided into two subsets: a control or verification set for which annotations are known, and an unknown set for which no verified annotations exist. The verification set is used to control against the tags provided for the unknown set. If the user provides correct verification tags, the tags for the unknown set are promoted. An image with a promoted tag must be validated by other users before it can be classed as annotated and added to the verification set. Given a partially annotated database, the images can be incrementally annotated over time. The TagCaptcha system is intended to replace traditional text-based CAPTCHA systems currently used for human verification on the web in the fight against spam.

Multi-view clustering is an important problem in information retrieval due to the abundance of data offering many perspectives and generating multi-view representations. We investigate in this short note a late fusion approach for multi-view clustering based on the latent modeling of cluster-cluster relationships. We derive a probabilistic multi-view clustering model outperforming an early-fusion approach based on multi-view feature correlation analysis.

This paper proposes a novel representation space for multimodal information, enabling fast and efficient retrieval of video data. We suggest describing the documents not directly by selected multimodal features (audio, visual or text), but rather by considering cross-document similarities relatively to their multimodal characteristics. This idea leads us to propose a particular form of dissimilarity space that is adapted to the asymmetric classification problem, and in turn to the query-by-example and relevance feedback paradigm, widely used in information retrieval. Based on the proposed dissimilarity space, we then define various strategies to fuse modalities through a kernel-based learning approach. The problem of automatic kernel setting to adapt the learning process to the queries is also discussed. The properties of our strategies are studied and validated on artificial data. In a second phase, a large annotated video corpus, (ie TRECVID-05), indexed by visual, audio and text features is considered to evaluate the overall performance of the dissimilarity space and fusion strategies. The obtained results confirm the validity of the proposed approach for the representation and retrieval of multimodal information in a real-time framework.

This paper addresses the problem of matching visual parts of video sequences from within a large collection. The visual content of a video sequence is described by the set of the most representative local features extracted in its frames. An index is proposed that permits to retrieve efficiently the pre-computed distances between every sequences of the collection and every local features. Relying on this structure, the partial matching is performed through an interactive feature selection algorithm that iteratively integrates the user knowledge to estimate a model of the queried pattern. We show that the method performs well both in terms of retrieval accuracy and response time efficiency, on large video collections.

Motion transparency phenomena in image sequences are frequent, but classical methods of motion estimation are unable to deal with them. This paper describes a method for estimating optical flow by a generalization of the brightness constancy assumption to additive transparencies. The brightness constancy assumption is obtained by setting constant velocity fields during three images of a sequence. Thus, by a Taylor development to its second order, we reach an extension of the optical flow constraint equation. Since the equation is nonlinear, the Levenberg-Marquardt algorithm is used. In order to suppress the unavoidable aperture problem, a global model based on B-spline basis functions is applied with the aim of constraining optical flows. This description of motion allows us to work on a coarse to fine estimation of artificial image sequences that shows good convergence properties. It is also applied to natural image sequences.

This paper describes an approach for video shot detection. It relies on the temporal analysis of frame to frame optical flow measurements. These measures, which are motion wavelet coefficients directly estimated from the image sequence, exhibit discontinuities when shot boundaries occur. These transitions are detected by considering the temporal trajectories of the linear prediction errors of wavelet coefficients. Experimental results and performance evaluation using videos with reference segmentations are presented to demonstrate the efficiency of the approach.