Automatic TV commercial categorization is the key component of a smart commercial content analysis and understanding system. In this paper, we focus our research on commercial categorization by mining the temporal-spatial context of TV commercial. To boost the discrimination ability of the traditional VBoW in commercial categorization, a more flexible representation, i.e. sparse coding based VBoW (sVBoW), is presented to describe the co-occurrence of semantic units in different kinds of commercials. Meanwhile, both the temporal and spatial cues are fully exploited to reflect the context characteristics of TV commercial. The promising experimental results show the effectiveness of the proposed scheme on commercial categorization.

Automatic TV commercial block detection (CBD) and commercial block segmentation (CBS) are two key components of a smart commercial digesting system. In this paper, we focus our research on CBD and CBS by the means of collaborative exploitation of visual-audio-textual characteristics embedded in commercials. Rather than utilizing exclusively visual-audio characteristics like most previous works, an abundance of textual characteristics associated with commercials are fully exploited. Additionally, Tri-AdaBoost, an interactive ensemble learning manner, is proposed to form a consolidated semantic fusion across visual, audio, and textual characteristics. In order to segment a detected commercial block into multiple individual commercials, additional informative descriptors including textual characteristics are introduced to boost the robustness in the detection of frame marked with product information (FMPI). Together with the characteristics of audio spectral variation pointer and silent position, FMPI can provide a kind of complementary representation architecture to model the similarity of intra-commercial and the dissimilarity of inter-commercial. Experiments are conducted on a large video dataset from both China central television (CCTV) channels and TRECVID'05, and promising experimental results show the effectiveness of the proposed scheme.

Automatic TV commercial detection has become an indispensable part of content-based video analysis technique due to the explosive growth in TV commercial volume. In this paper, a multi-modal (i.e. visual, audio and textual modalities) commercial digesting scheme is proposed to alleviate two challenges in commercial detection, which are the generation of mid-level semantic descriptor and the application of effective discrimination method. Compared with the general program, some unique semantic characteristics are purposely embedded in the commercial to grasp more attention from audience. Aiming at exploring the power of these semantic characteristics, a kind of novel commercial-oriented descriptor from textual modality is proposed, besides taking advantage of those commonly used description means in light of audio and visual modalities. To boost the ability of discrimination of commercial from general program in multi-modal representation space, Tri-AdaBoost, a self-learning method by an interactive way across multiple modalities, is introduced to form a final consolidated decision for discrimination. Moreover, a heuristic post processing strategy based on the temporal consistency is taken to further reduce the false alarms. The promising experimental results show the effectiveness of the proposed scheme with respect to large video data collections.

Analysis on click-through data from a very large search engine log shows that users are usually interested in the top-ranked portion of returned search results. Therefore, it is crucial for search engines to achieve high accuracy on the top-ranked documents. While many methods exist for boosting video search performance, they either pay less attention to the above factor or encounter difficulties in practical applications. In this paper, we present a flexible and effective reranking method, called CR-Reranking, to improve the retrieval effectiveness. To offer high accuracy on the top-ranked results, CR-Reranking employs a cross-reference (CR) strategy to fuse multimodal cues. Specifically, multimodal features are first utilized separately to rerank the initial returned results at the cluster level, and then all the ranked clusters from different modalities are cooperatively used to infer the shots with high relevance. Experimental results show that the search quality, especially on the top-ranked results, is improved significantly.