Associating texts and images is an easy and intuitive task for a human being, but it raises some issues if we want that task to be accomplished by a computer. Among these issues, there is the problem of finding a common representation for images and sentences. Based on recent research about capsule networks, we define a novel model to tackle that issue. This model is trained and compared to other recent models on the Flickr8k database on Image Retrieval and Image Annotation (or Sentence Retrieval) tasks. We propose a new recurrent architecture inspired from capsule networks to replace the traditional LSTM/GRU and show how it leads to improved performances. Moreover, we show that the interest of our model goes beyond its performances and includes its intrinsic characteristics, which can explain why it performs particularly well on the Image Annotation task. In addition, we propose a routing procedure between capsules which is fully learned during the training of our model.

This paper presents investigations about the automatic identification of video genre by audio channel analysis. Genre refers to editorial styles such commercials, movies, sports... We propose and evaluate some methods based on both low and high level descriptors, in cepstral or time domains, but also by analyzing the global structure of the document and the linguistic contents. Then, the proposed features are combined and their complementarity is evaluated. On a database composed of single-stories web-videos, the best audio-only based system performs 9% of Classification Error Rate (CER). Finally, we evaluate the complementarity of the proposed audio features and video features that are classically used for Video Genre Identification (VGI). Results demonstrate the complementarity of the modalities for genre recognition, the final audio-video system reaching 6% CER.

We present an approach to video concept detection by building binary trees partitioning the label space, using visual and semantic similarity for multi-label datasets. The technique overcomes sparse annotations problem by increasing the number of positive examples per concept with the number of classifiers per concept, though sub-optimal, augmented too. We draw similarities between the proposed tree generation approach and Error Correcting Output Codes (ECOC) for multi-label classification and build ranked lists of video shots using weighted decoding or weighted tree traversal. We build a set of different trees based on the presented criterion each partitioning the label space in its own specific way. Inspired by the work in [1] we amass information from ensemble of trees to build the final ranked list, but using a different criterion. The classification resulting in ensemble error correction is complementary to One-vs-All classification and increases concept detection performance significantly on the TRECVID 2010 and 2013 datasets.

Research interest in cotraining is increasing which combines information from usually two classifiers to iteratively increase training resources and strengthen the classifiers. We try to select classifiers for cotraining when more than two representations of the data are available. The classifier based on the selected representation or data descriptor is expected to provide the most complementary information as new labels for the target classifier. These labels are critical for the next learning iteration. We present two criteria to select the complementary classifier where classification results on a validation set are used to calculate statistics for all the available classifiers. These statistics are used not only to pick the best classifier but also ascertain the number of new labels to be added for the target classifier. We demonstrate the effectiveness of classifier selection for semantic indexing task on the TRECIVD 2013 dataset and compare it to the self-training.

In this article, we propose a novel Hamming embedding kernel with informative bag-of-visual words to address two main problems existing in traditional BoW approaches for video semantic indexing. First, Hamming embedding is employed to alleviate the information loss caused by SIFT quantization. The Hamming distances between keypoints in the same cell are calculated and integrated into the SVM kernel to better discriminate different image samples. Second, to highlight the concept-specific visual information, we propose to weight the visual words according to their informativeness for detecting specific concepts. We show that our proposed kernels can significantly improve the performance of concept detection.

Visual concept detection is a very active field of research, motivated by the increasing amount of digital video available. While most systems focus on the processing of visual features only, in the context of internet videos other metadata is available which may provide useful information. In this paper, we investigate the role of the uploader information, the person who uploaded the video. We propose a simple uploader model which includes some knowledge about the content of videos uploaded by a given user. On the TRECVID 2012 Semantic Indexing benchmark [1], we show that this simple model is able to improve the concept detection score of all the 2012 participants, even the best ones, by only re-ranking the proposed shots. We also present some statistics which show that even though most TRECVID systems are based on visual features only, they provide results which are biased in favor of test videos for which the uploader was present in the development data. This work suggests further research on the use of metadata for visual concept detection, and a different way of organizing benchmark data to assess the visual performance of detectors.

Local Image Descriptors (LID) aggregation models such as Bag of Words and Fisher Vectors represent an image based on the distribution of its LIDs given a global model, e.g. a visual codebook or a Gaussian Mixture.

Marginal Alphabets (MEDA) were proposed as an alternative to Bag of Words (BoW) for image representation. They aggregate sets of locally extracted descriptors (LEDs) by using visual alphabets based on the marginal approximation of the LED components. Compared to the exponential complexity of the BoW codebooks, the MEDA model is very efficient because each dimension of the LED is quantized independently. However, MEDA lacks of considering the relations between the LED components, loosing precious information for image representation.

In this paper we describe a system that automatically extracts appealing scenes from a set of broadcasting videos. Unlike traditional computational aesthetic models that try to predict the hardly measurable degree of "beauty", we chose to build a system that retrieves "interesting" scenes. We create a training database of Flickr images annotated with their corresponding Flickr "interestingness" degree. We then extract existing and novel aesthetic/semantic features from the training set. Based on such features, we build a graded-relevance "interestingness" model and we rank the test shots according to their predicted "interestingness".

This paper addresses the problem of unsupervised TV programs structuring. Program structuring allows direct and non linear access to the desired parts of a program. Our work addresses the structuring of recurrent TV programs like news, entertainment programs, TV shows, TV magazines. In a previous work we proposed a program structuring method based on the detection of video recurrences. In this paper we extend our study to audio recurrences and verify their influence on the final structuring. We evaluate the structuring results on both approaches (audio and video) separately and jointly. We use for evaluation a 62 hours dataset corresponding to 97 episodes of TV programs.

In this paper we present Saliency Moments, a new, holistic descriptor for image recognition inspired by two biological vision principles: the gist perception and the selective visual attention. While traditional image features extract either local or global discriminative properties from the visual content, we use a hybrid approach that exploits some coarsely localized information, i.e. the salient regions shape and contours, to build a global, low-dimensional image signature. Results show that this new type of image description outperforms the traditional global features on scene and object categorization, for a variety of challenging datasets. Moreover, we show that, when combined with other existing descriptors (SIFT, Color Moments, Wavelet Feature and Edge Histogram), the saliency-based features provide complementary information, improving the precision of a retrieval system we build for the TRECVID 2010.

Currently there are a lot of algorithms for video summarization; however most of them only represent visual information. In this paper, we propose two approaches for the construction of the summary using both video and text. One approach focuses on static summaries, where the summary is a set of selected keyframes and keywords, to be displayed in a fixed area. The second approach addresses dynamic summaries where video segments are selected based on both their visual and textual content to compose a new video sequence of predefined duration. Our approaches rely on an existing summarization algorithm, Video Maximal Marginal Relevance (Video-MMR), and its extension Text Video Maximal Marginal Relevance (TV-MMR) proposed by us. We describe the details of those approaches and present experimental results.

Bag of Words (BOW) models are nowadays one of the most effective methods for visual categorization. They use visual dictionaries to aggregate the set of local descriptors extracted from a given image. Despite their high discriminative ability, one of the major drawbacks of BOW still remains the computational cost of the visual dictionary, built by clustering in the high dimensional feature space. In this paper we introduce a fast, effective method for local image descriptors aggregation that is based on marginal approximations, i.e. the approximation of each descriptor component distribution. We quantize each dimension of the feature space, obtaining a visual alphabet that we use to map the image descriptors in a fixed-length visual signature. Experimental results show that our new method outperforms the traditional BOW model in both accuracy and efficiency for the scene recognition task. Moreover, we discover that the marginal-based aggregation provides complementary information with respect to BOW, by combining the two models in a video retrieval system based on TRECVID 2010.

Video Summarization has become an important tool for multimedia information processing, but the automatic evaluation of a video summarization system remains a challenge. A major issue is that an ideal "best" summary does not exist, although people can easily distinguish "good" from "bad" summaries. A similar situation arise in machine translation and text summarization, where specific automatic procedures, respectively BLEU and ROUGE, evaluate the quality of a candidate by comparing its local similarities with several human-generated references. These procedures are now routinely used in various benchmarks. In this paper, we extend this idea to the video domain and propose the VERT (Video Evaluation by Relevant Threshold) algorithm to automatically evaluate the quality of video summaries. VERT mimics the theories of BLEU and ROUGE, and counts the weighted number of overlapping selected units between the computer-generated video summary and several human-made references. Several variants of VERT are suggested and compared.

During the last years, the development of rushes video summarization systems has greatly increased thanks to the international evaluation campaign TRECVID. In this paper, we propose an automation of the manual TRECVID evaluation using machine learning techniques. We train an automatic assessor to perform evaluation on summary content and we show a high correlation between the manual evaluation performed in TRECVID 2008 and our automatic assessor.

Most previous works on video summarization target on a single video document. With the popularity of video corpus (e.g. news video archives) and Web videos, video article that consists of a set of relevant videos are frequently confronted by users. By the traditional single-document summarization, these videos are treated independently and the results are usually redundant due to the lack of inter-video analysis. To efficiently manage video articles, in this paper, we propose an approach for multi-document video summarization by exploring the redundancy between different videos. The importance of keyframes is first measured by the content inclusion based on intra- and inter-video similarities. We then propose a minimum description length (MDL) for automatically determining the appropriate length of the summary. Finally a video summary is generated for users to browse the content of the whole video article. We show that multi-document video summarization presents more elegant and informative summaries compared with single-document approach.

Video summarization is a useful tool which allows a user to grasp rapidly the essence of a video. In the development of this research topic we propose a new method based on different individual content segmentation and selection tools in a collaborative system. The main innovation of this work is to merge results from different approaches, so as to benefit from their respective qualities. Our system is organized in two phases: first segmentation of the video, second identification of relevant and redundant segments. The final list of selected segments is used to concatenate the video segments and build the final summary. In order to assess the effectiveness of this organization, we evaluate our system with a method based on the TRECVID 2007 BBC rushes summarization evaluation pilot and compare our performance with existing systems.

This paper studies the problem of background/object differentiation in a keypoint-based tracking application where the object is delimited with a bounding box. We present a keypoint labeling algorithm based on four features: the label of the matched keypoint, color, motion, and position. We discuss methods to best exploit these features, then we detail our labeling algorithm and validate it with some experiments on several tracking video sequences.

In this demonstration, we present the Annotation Tool that is being developed in the porTiVity project to annotate video objects for Interactive Television programs. This tool includes various video processing components to structure and speed-up the annotation process, such as shot segmentation, key-frame extraction, object tracking and object redetection. A specific feature is that the tool includes a preprocessing phase where a quantity of information is precomputed, so that the annotation itself can be done quite rapidly.

In this paper, we propose a new type of multi-dimensional hidden Markov model based on the idea of dependency tree between positions. This simplification leads to an efficient implementation of the re-estimation algorithms, while keeping a mix of horizontal and vertical dependencies between positions. We explain DT-HMM and we present the formulas for the maximum likelihood re-estimation. We illustrate the algorithm by training a 2-dimensional model on a set of coherent images

In this paper, we study the problem of the fast selection of video objects, as an aid for the efficient semi-automatic annotation of video programs. In a regular system, the user has to draw a bounding box around the object, requiring at least two clicks on the image. We propose and experiment algorithms that allow the selection by indicating only one point inside the object, therefore requiring only one click. The problem is then to identify the correct bounding box. We use an attention model and a growing algorithm to construct the most plausible bounding box, based on the comparison of the interior, the border and the outside of the box. We present some experimentation that suggests that in many cases, our algorithm is able to propose a reasonable bounding box

The GMF4iTV project (Generic Media Framework for Interactive Television) is an IST European project that developed an end-to-end broadcasting platform providing interactivity on heterogeneous multimedia devices such as Set-Top-Boxes, PCs and PDAs according to the Multimedia Home Platform (MHP) part of the DVB standard. The developed platform allows the content providers to create enhanced audiovisual contents with a degree of interactivity at moving object level or shot changes in a video. The end user is then able to interact with moving objects from the video or individual shots allowing the enjoyment of additional contents associated to them (MHP applications, HTML pages, JPEG, MPEG-4 files,...).

The GMF4iTV project is building an end-to-end broadcast system for providing interactivity to TV programs through active video objects. This paper describes the personalization scheme that has been defined and implemented in this project. On the authoring side, an interface tool allows the video producers to annotate both objects and associated additional content, with attributes from different ontologies. On the receiver side, a personalization engine compares these attributes with the user profile to personalize the interaction with the user. We describe the various components of the system and provide some application examples.

Low-level features are now becoming insufficient to build efficient content-based retrieval systems. The interest of users is not any more to retrieve visually similar content, but they expect retrieval systems to find documents with similar semantic content. Bridging the gap between low-level features and semantic content is a challenging task necessary for future retrieval systems. Latent semantic indexing (LSI) was successfully introduced to efficiently index text documents. We propose to adapt this technique to efficiently represent the visual content of video shots for semantic content detection. Although we restrict our approach to visual features, it can be extended with minor changes to audio and motion features to build a multi-modal system. The semantic content is then detected thanks to two classifiers: k-nearest neighbors and neural network classifiers. Finally, in the experimental section we show the performances of each classifier and the performance gain obtained with LSI features compared to traditional features.

This paper presents the evaluation of a number of algorithm alternatives for content based retrieval from a database of technical drawings representing patents. The objective is to help patent evaluators in their quest for a possible patent bearing too much similarity with the one under investigation. To achieve this, we have devised a system where images (drawings) are represented using attributed graphs based on the extracted line-patterns or histograms of attributes computed from the graphs. Retrieval is either performed using histogram comparison (see Huet, B. and Hancock, E.R., PAMI, vol.21, no.12, p.1363-70, 1999) or thanks to a graph similarity measure (see Huet and Hancock, CVPR, p.138-43, 1998). Promising results are presented along with possible work extension.

The growing availability of multimedia data such as video on personal computers and home equipment creates a strong requirement for efficient tools to manipulate this type of data. Automatic summarization is one of these tools, which automatically creates a short version or subset of key-frames which contains as much information as possible as in the original video. Several approaches have been proposed to define and identify what is the most important content in a video. In this paper, we propose a new approach for the automatic creation of summaries for multi-episode videos, such as TV series. In this case, it is necessary to identify similarities and differences among videos (what’s common, what’s unique, how they differ) in order to find which elements best characterize a particular video with respect to the others. We describe our proposed method, provide the results of its application on a sample set of videos, and suggest a new criterion to evaluate the quality of the summaries that have been created.

This paper presents a comparison of some methodologies for the automatic construction of video summaries. The work is based on the simulated user principle to evaluate the quality of a video summary in a way which is automatic, yet related to user's perception. The method is studied for the case of multi-episode video, where we do not only describe what is important in a video, but rather what distinguishes this video from the others. Experimental results are presented to support the proposed ideas.

The information globalization induced by the rapid development of the Internet and the accompanying adoption of the Web throughout the society leads to Websites which reach large audiences. The diversity of the audiences and the need of customer retention require active Websites, which expose themselves in a customized or personalized way: We call those sites User-adapted Websites. New technologies are necessary to personalize and customize content. Information filtering can be used for the discovery of important content and is therefore a key-technology for the creation of user-adapted Websites. In this article, we focus on the application of collaborative filtering for user-adapted Websites. We studied techniques for combining and integrating content-based filtering with collaborative filtering in order to address typical problems in collaborative filtering systems and to improve the performance. Other issues which are mentioned but only lightly covered include user interface challenges. To validate our approaches we developed a prototype user-adapted Website, the Active Web-Museum, a museum Website, which exposes its collection in a personalized way by the use of collaborative filtering.

While one-dimensional hidden Markov models have been very successfully applied to numerous problems, their extension to two dimensions has been shown to be exponentially complex, and this has very much restricted their usage for problems such as image analysis. In this paper we propose a novel algorithm which is able to approximate the search for the best state path (Viterbi decoding) in a 2D HMM. This algorithm makes certain assumptions which lead to tractable computations, at a price of loss in full optimality. We detail our algorithm, its implementation, and present some experiments on handwritten character recognition. Because the Viterbi algorithm serves as a basis for many applications, and 1D HMMs have shown great flexibility in their usage, our approach has the potential to make 2D HMMs as useful for 2D data as 1D HMMs are for 1D data such as speech.

The Internet is evolving from a static collection of hypertext, to a rich assortment of dynamic services and products targeted at millions of Internet users. For most sites it is a crucial matter to keep a close tie between the users and the site.

In this paper, we study the automatic construction of personalized TV News programs, where we want to build a program with predefined duration and maximum content value for a specific user. We combine video indexing techniques to parse TV News recordings into stories, and information filtering techniques to select stories which are most adequate given the user profile. We formalize the selection process as an optimization problem, and we study how to take into account duration in the selection of stories. Experiments show that a simple heuristic can provide high quality selection with little computation. We also describe two prototypes, which implement two different mechanisms for the construction of user profiles:

A theorem which shows that the local maximum found by the forward-backward algorithm in the case of discrete hidden Markov models is really local is presented. By this it is meant that this local maximum is restricted to lie in the same connected component of the set (x:P(x)>or=P(x/sub 0/)) as the initial point x/sub 0/ (where P(x) is the polynomial being maximized). This theoretical result suggests that, in practice, the choice of the initial point is important for the quality of the maximum obtained by the algorithm.>

The authors describe a series of experiments in which the phonetic baseform is deduced automatically for new words by utilizing actual utterances of the new word in conjunction with a set of automatically derived spelling-to-sound rules. Recognition performance was evaluated on new words spoken by two different speakers when the phonetic baseforms were extracted via the above approach. The error rates on these new words were found to be comparable to or better than when the phonetic baseforms were derived by hand, thus validating the basic approach.>

Experiments on the use of a probabilistic model to tag English text, that is, to assign to each word the correct tag (part of speech) in the context of the sentence, are presented. A simple triclass Markov model is used, and the best way to estimate the parameters of this model, depending on the kind and amount of training data that is provided, is found. Two approaches are compared: the use of text that has been tagged by hand and comparing relative frequency counts; and use text without tags and training the model as a hidden Markov process, according to a maximum likelihood principle. Experiments show that the best training is obtained by using as much tagged text as is available, a maximum likelihood training may improve the accuracy of the tagging.>

The authors experiment with a combination of two methods previously proposed to improve the performance of their speech recognition system. One method is based on the definition of an improved system of phonetic units, which takes into account the most important coarticulation effects. This system has been defined using knowledge about coarticulation, and by studying the errors of a standard phonetic system. The second method is based on the use of maximum mutual information (MMI) as a criterion in the training phase of the speech recognition system. MMI is designed to maximize the probability of the correct text versus the other possible texts, and it is expected to provide better discrimination of the correct text than the standard maximum-likelihood criterion. These methods have been tested independently on phonetic recognition, and each of them improved the recognition accuracy of the system. Results of recognition experiments that combine the two methods are presented and discussed. They show that this combination improves the average recognition rate, both for phonetic and word recognition.>

The application of maximum-mutual-information (MMI) training to hidden Markov models (HMMs) is studied for phonetic recognition. MMI training has been proposed as an alternative to standard maximum-likelihood (ML) training. In practice, MMI training performs better (produces models that are more accurate) than ML training. The fundamental notions of HMM, ML and MMI training are reviewed, and it is shown how MMI training can be applied easily to the case of phonetic models and phonetic recognition. Some computational heuristics are proposed to implement these computations practically. Some experiments (training and recognition) are detailed that show that the phonetic error rate decreases significantly when MMI training is used, as compared with ML training.>

This paper proposes a new strategy, the Multi-Level Decoding (MLD), that allows to use a Very Large Size Dictionary (VLSD, size more than 100,000 words) in speech recognition. MLD proceeds in three steps:\bulleta Syllable Match procedure uses an acoustic model to build a list of the most probable syllables that match the acoustic signal from a given time frame.\bulletfrom this list, a Word Match procedure uses the dictionary to build partial word hypothesis.\bulletthen a Sentence Match procedure uses a probabilistic language model to build partial sentence hypothesis until total sentences are found. An original matching algorithm is proposed for the Syllable Match procedure. This strategy is experimented on a dictation task of French texts. Two different dictionaries are tested,\bulletone composed of the 10,000 most frequent words,\bulletthe other composed of 200,000 words. The recognition results are given and compared. The error rate on words with 10,000 words is 17.3%. If the errors due to the lack of coverage are not counted, the error rate with 10,000 words is reduced to 10.6%. The error rate with 200,000 words is 12.7%.

An approach for supporting large vocabulary in speech recognition is to use broad phonetic classes to reduce the search to a subset of the dictionary. In this paper, we investigate the problem of defining an optimal classification for a given speech decoder, so that these broad phonetic classes are recognized as accurately as possible from the speech signal. More precisely, given Hidden Markov Models of phonemes, we define a similarity measure of the phonetic machines, and use a standard classification algorithm to find the optimal classification. Three measures are proposed, and compared with manual classifications.

This paper relates different kinds of language modeling methods that can be applied to the linguistic decoding part of a speech recognition system with a very large vocabulary. These models are studied experimentally on a pseudophonetic input arising from French stenotypy. We propose a model which combines the advantages of a statistical modeling with information theoretic tools, and those of a grammatical approach.

In this paper, we study the combination of an information theoretic tool (Markov modeling of natural language [3]) with probabilistic grammatical analysis. Continuous Speech Recognition for natural language raises a lot of difficulties, both for the acoustic processing and the linguistic decoding. Our work specifically concerns the linguistic decoding techniques for a very large (140,000 entries) French dictionary, and a oral open discourse. So the task is to transcribe a continuous string of pseudo-phonemes into written text. This string would be ideally the output of a perfect acoustic processor. We present a grammar designed for automatic transcription and compute probabilities for the rules. We compare its results with those obtained earlier with Markov modeling. We show that it is possible to combine the two approaches and get better results than each model separately.

In this paper, we study the complexity of the recognition process for a large vocabulary, with a specific application using a large French dictionary. In the first part, we review the lexical and grammatical differences between French and English that affect recognition complexity. Then we compute various measurements on a large pseudo-phonetic French dictionary. In the second part, we report experiments on a large real text. To measure recognition complexity, we compute the average branching factor arising in all possible spellings of the phonetic transcription of the text, according to our dictionary. We show that a Markov modeling of French allows us to reduce significantly the branching factor by discarding improbable choices.

Lip-reading is widely used by profoundly deaf individuals for the reception of the spoken language. This is a very difficult task because the labial image is ambiguous, The final goal of this work is to provide the deaf person with additional information (or "keys") which disambiguate the labial image. Phoneme recognition in continuous speech is used to produce the keys. To allow complete freedom in running speech, and in order to provide keys synchronously with speech production, no lexical, syntactical or semantical informations are used. Algorithms are adapted to a given speaker through a learning phase where prototypes are built for the phonetic units to be recognized. Recognition algorithms are a combination of segmentation and centisecond labeling. The keys system is optimized taking into account the confusions made by the recognition programs. Recognition scores for multiple speakers are indicated both at the phonetic level and at the keys level.