Intelligent video surveillance system is built to automatically detect events of interest, especially on object tracking and behavior understanding. In this paper, we focus on the task of human action recognition under surveillance environment, specifically in a multi-camera monitoring scene. Despite many approaches have achieved success in recognizing human action from video sequences, they are designed for single view and generally not robust against viewpoint invariant. Human action recognition across different views remains challenging due to the large variations from one view to another. We present a framework to solve the problem of transferring action models learned in one view (source view) to another view (target view). First, local space-time interest point feature and global shape-flow feature are extracted as low-level feature, followed by building the hybrid Bag-of-Words model for each action sequence. The data distribution of relevant actions from source view and target view are linked via a cross-view discriminative dictionary learning method. Through the view-adaptive dictionary pair learned by the method, the data from source and target view can be respectively mapped into a common space which is view-invariant. Furthermore, We extend our framework to transfer action models from multiple views to one view when there are multiple source views available. Experiments on the IXMAS human action dataset, which contains videos captured with five viewpoints, show the efficacy of our framework.

This paper proposes a novel multi-view human action recognition method by discovering and sharing common knowledge among different video sets captured in multiple viewpoints. To our knowledge, we are the first to treat a specific view as target domain and the others as source domains and consequently formulate the multi-view action recognition into the cross-domain learning framework. First, the classic bag-of-visual word framework is implemented for visual feature extraction in individual viewpoints. Then, we propose a cross-domain learning method with block-wise weighted kernel function matrix to highlight the saliency components and consequently augment the discriminative ability of the model. Extensive experiments are implemented on IXMAS, the popular multi-view action dataset. The experimental results demonstrate that the proposed method can consistently outperform the state of the arts.

We introduce a novel query-to-modality mapping approach to the TRECVid 2010 known-Item video search (KIS) task. To search for a specific target video, a KIS query is verbose with many multi-modal attributes. Issuing all search terms to a retrieval engine will confuse the search criteria in different modalities and result in "topic drift". We propose decomposing a KIS query into a set of short uni-modal subqueries and issue them to the search index of the corresponding modality features, such as text-based metadata, visualbased high-level features. To do so, we introduce novel syntactic query features and cast the query-to-modality mapping as a classification problem. Retrieval results on the TRECVid 2010 KIS dataset shows that our approach outperforms existing methods by a significant margin.

In this paper, we propose a framework to create a unique signature for a video. The framework aims to enable copy detection for online video sharing. This work improves on our earlier work by incorporating several features. First, we introduce a Singular Value Decomposition (SVD) method to detect shot changes and extract meaningful keyframes. Second, we incorporate a novel two-level bitmap structure for indexing the video signatures. Third, instead of Euclidean distance matching, a Boolean operation is used for video matching. The experimental result shows that the overall system performance improves as compared to previous work. In addition, the system is demonstrated to be effective and robust to several transformations such as color shift, brightness change, the addition of Gaussian noise as well as modified video with inserted text or logo.

Leveraging on the strength of energy-based processing for transient detection and pitch-based processing for softer onsets detection, we present a system that combines both energy and pitch cues for detecting onsets from different instrument categories. Given an audio input from an arbitrary instrument category, the system performs preliminary onset categorization based on the general note characteristics and then performs onsets integration based on the categorization. In addition, the proposed pitch processing technique explores musically relevant features extracted from the chromagram, which are robust for detecting pitch changes. The proposed system showed good detection performance on the MIREX audio onset detection dataset.

In this paper, we propose a framework for detecting near duplicate copies of a video based on an ordinal method. The framework also incorporates a bitmap indexing structure instead of a typical indexing structure used in the previous published work. Using this method, two levels of indices are constructed. The first level of this process groups each input video (represented by their key frames) into k clusters. These clusters and the associated key frames are then used to construct the first level index. The second level of this process converts ordinal-based video signatures (generated using the technique developed in earlier work) into bitmap vectors. By adopting this two-level indexing scheme, query processing times are significantly reduced. This is because, the system is required to match only videos in the clusters that are relevant to the query and not all the videos in the database. Additionally, the technique implemented utilizes a bitmap structure for indexing, resulting in less storage space. Furthermore, we are able to employ low-cost Boolean operations such as AND, OR, and XOR in the matching process instead of Euclidean distance or other similar matching algorithms. This helps to reduce the computational time for video matching. The system has demonstrated to effectively reduce the space needed to store collections of video signatures in a database, as well as improving the overall system performance. In addition, initial results show that the system is effective and robust to several transformations such as changes in brightness, color, contrast, resolution (reduction) as well as the addition of noise.

In this paper, a method for detecting infringements or modifications of a video in real-time is proposed. The method first segments a video stream into shots, after which it extracts some reference frames as keyframes. This process is performed employing a Singular Value Decomposition (SVD) technique developed in this work. Next, for each input video (represented by its keyframes), ordinal-based signature and SIFT (Scale Invariant Feature Transform) descriptors are generated. The ordinal-based method employs a two-level bitmap indexing scheme to construct the index for each video signature. The first level clusters all input keyframes into k clusters while the second level converts the ordinal-based signatures into bitmap vectors. On the other hand, the SIFT-based method directly uses the descriptors as the index. Given a suspect video (being streamed or transferred on the Internet), we generate the signature (ordinal and SIFT descriptors) then we compute similarity between its signature and those signatures in the database based on ordinal signature and SIFT descriptors separately. For similarity measure, besides the Euclidean distance, Boolean operators are also utilized during the matching process. We have tested our system by performing several experiments on 50 videos (each about 1/2 hour in duration) obtained from the TRECVID 2006 data set. For experiments set up, we refer to the conditions provided by TRECVID 2009 on "Content-based copy detection" task. In addition, we also refer to the requirements issued in the call for proposals by MPEG standard on the similar task. Initial result shows that our framework is effective and robust. As compared to our previous work, on top of the achievement we obtained by reducing the storage space and time taken in the ordinal based method, by introducing the SIFT features, we could achieve an overall accuracy in F1 measure of about 96% (improved about 8%).

In this paper, we present a framework called ldquoWhat-you-seeis- what-I-see (WYSIWIS)rdquo. The framework describes a new trend for multimedia information (such as images and videos) sharing using an Eyetap device and a method to retrieve similar multimedia content. This Eyetap device is equipped with a video camera and networking capability. It can be used to capture an image or recording a video of a place being visited, sending it to the server to get the information about this place (through search mechanism), the captured image/video is then stored into the server for sharing. With this function, mobile users can also share the capturing video in real-time. This is similar to mobile phone equipped with video conference function. However, it is more convenient as users just wear this Eyetap device as normal glasses and performs their daily routine while stay connected.

The development of effective music retrieval and recommendation applications requires meaningful features for characterizing music audio. The rhythm of music audio in particular, besides timbre and melody, is essential in describing the music piece. This paper illustrates how drum loop patterns characterize the salient rhythm structure of music and an approach for drum loop pattern extraction is described. The extracted drum loop patterns are formed based on the temporal regularity (accent on meter) and drum types (bass and snare) of the beats and are good representations of the rhythmic dimension of music. The extracted drum loop patterns are useful for music segmentation and further work would include music classification and similarity matching.

This paper proposes an integrated framework that combines intra-shot and temporal inter-shot sequence analysis based on visual features to find stable patterns for video annotation. At the shot level, we perform multi-stage kNN classification using the global visual features to identify good candidate shots containing the concept. At the sequence level, we aim to find patterns of shot sequences around candidate shots with consistent statistical characteristics and dynamics. We discretize the shot contents into fixed set of tokens, and transform the high dimensional continuous video streams into tractable token sequences. We then extend the soft matching model to reveal video sequence patterns and flexibly match the patterns around candidate shots. We combine both local shot matching method and generalized pattern model using both visual and text features. Experimental results on TRECVID2006 dataset demonstrate that the proposed approach is effective.

Global rule induction technique has been successfully used in information extraction (IE) from text documents. In this paper, we employ the technique to identify story boundaries in news video. We divide our framework into two levels: shot and story levels. We use a hybrid algorithm to classify each input video shot into one of the predefined genre types and employ the global rule induction technique to extract story boundaries from the sequence of classified shots. We evaluate our rule induction based system on ~120-hours of news video provided by TRECVID 2003. The results show that we could achieve an F1accuracy of over 75%

A multi-modal two-level framework for news story segmentation was proposed in Chaisorn et al. (2002). This paper presents our extended work scaled to cope with a large news video corpus used in TRECVID 2003 evaluation. We divided our system into two levels: the shot level that classifies input video shots into one of the predefined categories using a hybrid of heuristic and learning based approaches; and story level that performs story segmentation using the HMM framework based on the output of shot level and other temporal features. A heuristic rule-based technique is then employed to classify each detected story into "news" or "miscellaneous". We evaluated our system on over 120 hours of news video and showed that our system could achieve an accuracy of more than 77%. Our system came first in the TRECVID 2003 story segmentation task

The segmentation of news video into story units is an important step towards effective processing and management of large news video archives. In the story segmentation task in TRECVID 2003, a wide variety of techniques were employed by many research groups to segment over 120-hour of news video. The techniques employed range from simple anchor person detector to soisticated machine learning models based on HMM and Maximum Entropy (ME) approaches. The general results indicate that the judicious use of multi-modality features coupled with rigorous machine learning models could produce effective solutions. This paper presents the algorithms and experience learned in TRECVID evaluations. It also points the way towards the development of scalable technology to process large news video corpuses.

When querying a news video archive, the users are interested in retrieving precise answers in the form of a summary that best answers the query. However, current video retrieval systems, including the search engines on the web, are designed to retrieve documents instead of precise answers. This research explores the use of question answering (QA) techniques to support personalized news video retrieval. Users interact with our system, VideoQA, using short natural language questions with implicit constraints on contents, context, duration, and genre of expected videos. VideoQA returns short precise news video summaries as answers. The main contributions of this research are: (a) the extension of QA technology to support QA in news video; and (b) the use of multi-modal features, including visual, audio, textual, and external resources, to help correct speech recognition errors and to perform precise question answering. The system has been tested on 7 days of news video and has been found to be effective.

The segmentation of news video into single-story semantic units is a challenging problem. This research proposes a two-level, multi-modal framework to tackle this problem. The video is analyzed at the shot and story unit (or scene) levels using a variety of features and techniques. At the shot level, we employ a decision tree to classify the shot into one of 13 predefined categories. At the scene level, we perform HMM (hidden Markov models) analysis to locate the story boundaries. We test the performance of our system using two days of news video obtained from the MediaCorp of Singapore. Our initial results indicate that we could achieve a high accuracy of over 95% for shot classification, and over 89% in F/sub 1/ measure on scene/story boundary detection.