Automatic medical image classification is difficult because of the lacking of training data. As manual labeling is too costly, we provide an automatic labeling solution to this problem by making use of the radiology report associated with the medical images. We first segment and reconstruct the 3D regions of interest (ROIs) from the medical images, and extract pathology and anatomy information from the associated report. We use an anatomical atlas to map the ROIs to the anatomy part(s) and match the pathology information of the same anatomy part(s) from the text. In this way, the ROIs are automatically labeled with pathology types which can be served as class labels, and a training data set of a large number of training instances is generated automatically. We extract the volume, color, location, and shape features of the ROIs, and classify the types of ROIs using these features. The overall evaluation result is promising to doctors and medical professionals. Our experiment is conducted using traumatic brain injury CT images; however, our framework of automatically labeling and classifying medical cases can be extended to medical images in other modality or of other anatomical part.

In spite of extensive studies on visual saliency, e.g., generating a saliency map from an image, less work has been addressed how to crop salient regions from saliency maps. We present a new approach to detect salient regions with maximum saliency density from videos. A branch-and-bound search algorithm is developed to find the global optimal solution efficiently. The proposed detection approach can automatically adapt to the shapes and motions of salient objects regardless of cluttered backgrounds. Moreover, by introducing an intermediate cropping window, video retargeting as an application of salient region detection gets optimized saliency coverage. Extensive experimental results validate the advantages of the proposed method.

Detection of salient objects in an image remains a challenging problem despite extensive studies in visual saliency, as the generated saliency map is usually noisy and incomplete. In this paper, we propose a new method to discover the salient object without prior knowledge on its shape and size. By searching the sub-image, i.e., a bounding box of maximum saliency density, the new formulation can automatically crop the salient objects of various sizes in spite of the cluttered background, and is capable to handle different types of saliency maps. A global optimal solution is obtained by the proposed density-based branch-and-bound search. The proposed method can apply to both images and videos. Experimental results on a public dataset of 5000 images show that our unsupervised detection approach is comparable to the state-of-the-art learning-based methods. Promising results are also observed in the salient object detection for videos with a good potential in video retargeting.

Computed tomography (CT) is used widely in traumatic brain injury diagnosis. One axial brain CT scan consists of multiple slices with different heights along the brain axial direction. Indexing of brain CT slices is to order the slices and align each individual slice onto the corresponding brain axial height, which is an important step in content-based image retrieval and computer-assisted diagnosis. Current existing methods for this indexing task are through the image registration techniques by registering 2D image slices onto a 3D brain atlas. In this paper, instead of using the registration methods, we propose a fast indexing method using anatomical feature classification. In our method, the brain CT scan is divided into 6 height levels along the axial direction so that slices in each level share similar anatomical structure. In this way, the indexing problem becomes a classification problem that one series of scan slices are to be classified into 6 classes. Experimental results show that the proposed method is effective and efficient.

Large number of medical images are produced daily in hospitals and medical institutions, the needs to efficiently process, index, search and retrieve these images are great. In this paper, we propose a pathology based medical image annotation framework using a statistical machine translation approach. After pathology terms and regions of interest (ROIs) are extracted from training text and images respectively, we use machine translation model IBM Model 1 to iteratively learn the alignment between the ROIs and the pathology terms and generate an ROI-to-pathology translation table. In testing phase, we annotate the ROI in the image with the pathology label of the highest probability in the translation table. The overall annotation results and the retrieval performance are promising to doctors and medical professionals.

Audio podcasting has been widely used by many online sites such as newspapers, web portals, journal, etc., to deliver audio content to users through download or subscription. Within 1 to 30 minutes long of one podcast story, it is often that multiple audio advertisements (ads) are inserted into and repeated, with each of a length of 5 to 30 seconds, at different locations. Based on knowledge of typical structures of podcast contents, this paper proposes a novel efficient advertisement discovery approach to identify and locate unknown ads from a large collection of audio podcasting. Two techniques: candidate region segmentation and sampling technique are employed to speed up the search. The approach has been tested over a variety of podcast contents collected from MIT Technology Review, Scientific American, and Singapore Podcast websites. Experimental results show that the proposed approach achieves detection rate of 97.5% with a significant computation saving as compared to existing state-of-the art methods.

In intracranial pathological examinations using CT scan, brain midline shift (MLS) is an important diagnostic feature indicating the pathological severity and patient's survival possibility. In this paper, we develop a new method of tracing the brain midline shift in traumatic brain injury (TBI) CT images using its original cause - the hemorrhage. Firstly, we model the relationship between the hemorrhage and the midline deformation caused by it using a linear regression model (H-MLS model). Secondly, using the H-MLS model, the deformed midline is predicted from the hemorrhage detected in CT images. Finally, the predicted deformed midline is adjusted according to the visual symmetry information. Preliminary experiments show that the proposed method is effective and time-efficient.

The construction of probabilistic liver atlases has received little attention in the past. Existing methods are based on landmarks and are sensitive to their choices and placements. We propose an iterative landmark-free method based on dense volumes to construct linear unbiased diffeomorphic probabilistic atlases from liver CT images. The linear averaging of the transformed images is set as the common target space followed by pairwise diffeomorphic registrations to warp all images to the target using a recent-proposed efficient deformation approach during each iteration cycle. Iterative pairwise registrations are directly used to handle possible large deformations without the need for an extra step to remove global deformations such as the use of affine transformations in traditional methods. Compared with those approaches estimating the unbiased atlas and the transformations groupwise simultaneously, the current method is more efficient. The efficiency and the convergence of our method have been demonstrated experimentally by validation using 25 CT liver sets.

Visual event detection in video streams allows easier access to, and better organization of large media collections. This paper presents an event detection framework with a novel feature that incorporates flow, appearance and trajectory information jointly. While previous event detection methods have been designed for understanding human behaviours where the camera is either static or with minimal motion, a more general approach is needed as real-life events are always subjected to fast camera motion and involve non-human dynamic objects. Inspired by the success of dense and overlapping orientation histograms in human detection, we build an event descriptor using orientation histograms augmented with feature point trajectory information. We put our system to test on tennis videos which have significant camera motion and multiple dynamic objects, and achieved good classification performance under a comparable setting.

We present a higher-level visual representation, visual synset, for object categorization. The visual synset improves the traditional bag of words representation with better discrimination and invariance power. First, the approach strengthens the inter-class discrimination power by constructing an intermediate visual descriptor, delta visual phrase, from frequently co-occurring visual word-set with similar spatial context. Second, the approach achieves better intra-class invariance power, by clustering delta visual phrases into visual synset, based their probabilistic dasiasemanticspsila, i.e. class probability distribution. Hence, the resulting visual synset can partially bridge the visual differences of images of same class. The tests on Caltech-101 and Pascal-VOC 05 dataset demonstrated that the proposed image representation can achieve good accuracies.

Medical text mining has gained increasing interest in recent years. Radiology reports contain rich information describing radiologistpsilas observations on the patientpsilas medical conditions in the associated medical images. However, as most reports are in free text format, the valuable information contained in those reports cannot be easily accessed and used, unless proper text mining has been applied. In this paper, we propose a text mining system to extract and use the information in radiology reports. The system consists of three main modules: a medical finding extractor, a report and image retriever, and a text-assisted image feature extractor. In evaluation, the overall precision and recall for medical finding extraction are 95.5% and 87.9% respectively, and for all modifiers of the medical findings 88.2% and 82.8% respectively. The overall result of report and image retrieval module and text-assisted image feature extraction module is satisfactory to radiologists.

Multi-slice computer tomography (CT) scans are widely used in todaypsilas diagnosis of head traumas. It is effective to disclose the bleeding and fractures. In this paper, we present an automated detection of CT scan slices which contain hemorrhages. Our method is robust towards various rotation, displacement and motion blur. Detection of these pathological slices will be useful for further diagnosis and retrieval.

Efficiency and robustness are the two most important issues for multiobject tracking algorithms in real-time intelligent video surveillance systems. We propose a novel 2.5-D approach to real-time multiobject tracking in crowds, which is formulated as a maximum a posteriori estimation problem and is approximated through an assignment step and a location step. Observing that the occluding object is usually less affected by the occluded objects, sequential solutions for the assignment and the location are derived. A novel dominant color histogram (DCH) is proposed as an efficient object model. The DCH can be regarded as a generalized color histogram, where dominant colors are selected based on a given distance measure. Comparing with conventional color histograms, the DCH only requires a few color components (31 on average). Furthermore, our theoretical analysis and evaluation on real data have shown that DCHs are robust to illumination changes. Using the DCH, efficient implementations of sequential solutions for the assignment and location steps are proposed. The assignment step includes the estimation of the depth order for the objects in a dispersing group, one-by-one assignment, and feature exclusion from the group representation. The location step includes the depth-order estimation for the objects in a new group, the two-phase mean-shift location, and the exclusion of tracked objects from the new position in the group. Multiobject tracking results and evaluation from public data sets are presented. Experiments on image sequences captured from crowded public environments have shown good tracking results, where about 90% of the objects have been successfully tracked with the correct identification numbers by the proposed method. Our results and evaluation have indicated that the method is efficient and robust for tracking multiple objects (>or= 3) in complex occlusion for real-world surveillance scenarios.

Widely used in the evaluation of retrieval systems, the pooling method collects top ranked images from submitted retrieval systems resulting in possibly a very large pool of images. Inevitably, the pool may contain outliers. Human experts then manually annotate the relevance of them to create a ground truth for evaluation. Studies show that this annotation is time-consuming, tedious and inconsistent. To reduce human workload, this paper introduces an automatic method to detect outliers. Different from traditional detection methods using unsupervised techniques only, we utilize both supervised and unsupervised techniques sequentially as both positive and negative examples are (partially) available in this context. Specifically, support vector machines (SVMs) and fuzzy c-means clustering are used to predict data relevance and "outlierness". Performance improvements using our method after outlier removal have been validated on the medical image retrieval task in ImageCLEF 2004.

In this paper we propose an efficient and robust method to automatically discover unknown short video repeats with arbitrary lengths, from a few seconds to a few minutes, from large video databases or streams. The proposed method consists of non-uniform video segmentation, self-similarity analysis, locality sensitive hashing, and video repeat boundary refinement. In order to achieve efficient and accurate processing feature extraction and similarity measure are performed at two levels: video frame level and video segment level. Experiments are conducted on 12 hour CNN/ABC news, and 12 hour documentaries (Discovery and National Geography), high recall and precision of 98% - 99% have been achieved. Video repeats' boundaries can be located within several frames. Applying the proposed method for video structure analysis is also briefly discussed.

In this paper, we propose a novel automatic approach for personalized music sports video generation. Two research challenges are addressed, specifically the semantic sports video content extraction and the automatic music video composition. For the first challenge, we propose to use multimodal (audio, video, and text) feature analysis and alignment to detect the semantics of events in broadcast sports video. For the second challenge, we introduce the video-centric and music-centric music video composition schemes and proposed a dynamic-programming based algorithm to perform fully or semi-automatic generation of personalized music sports video. The experimental results and user evaluations are promising and show that our systems generated music sports video is comparable to professionally generated ones. Our proposed system greatly facilitates the music sports video editing task for both professionals and amateurs

This paper aims at repeat clip mining and knowledge discovery from video data. A unified approach is proposed to detect both unknown video repeats and known video clips of arbitrary length. Two detectors in a cascade structure are employed to achieve fast and accurate detection, and a reinforcement learning approach is adopted to efficiently maximize detection accuracy. In this approach very short video repeats (<1 s) and long ones can be detected by a single process, while overall accuracy remains high. Since video segmentation is essential for repeat detection, performance analysis is also conducted for several segmentation methods. Furthermore we propose a method to analyze video syntactical structure based on short video repeats detection. Experimental results on news videos demonstrate that identifying short video repeats is an effective way for video structure discovery and syntactical segmentation

A novel content-adaptive music watermarking technique is proposed in this article. To optimally balance inaudibility and robustness when embedding and extracting watermarks, the embedding scheme is highly related to the music structure and human auditory system (HAS). A note-based segmentation method is proposed and used for music vocal/instrumental boundary detection. A multiple bit hopping and hiding scheme with different embedding parameters is applied to vocal and instrumental frames of the music. The experimental results in inaudibility and robustness are provided to support all novel features in the proposed watermarking scheme.

In this paper, a multi-class classification system is developed for medical images. We have mainly explored ways to use different image features, and compared two classifiers: principle component analysis (PCA) and supporting vector machines (SVM) with RBF (radial basis functions) kernels. Experimental results showed that SVM with a combination of the middle-level blob feature and low-level features (down-scaled images and their texture maps) achieved the highest recognition accuracy. Using the 9000 given training images from ImageCLEFOS, our proposed method has achieved a recognition rate of 88.9% in a simulation experiment. And according to the evaluation result from the ImageCLEFOS organizer, our method has achieved a recognition rate of 82% over its 1000 testing images

In this paper, we propose an approach for TV commercial video classification by the categories of advertised products or services (e.g. automobiles, healthcare products, etc). Since automatic speech recognition (ASR) and optical character recognition (OCR) can deliver meaningful textual information related to products or services, TV commercial video classification is formulated as the problem of text categorization. However, there exist two challenges. Firstly, the background music of TV commercials makes ASR techniques yield erroneous and deficient output transcripts. Secondly, even if ASR and OCR could work perfectly, the limited textual information from TV commercials do not suffice to train a generic and non-overfitting text categorizer. For the first issue, our approach resorts to the external resources to expand deficient ASR and OCR transcripts. The output transcripts of ASR and OCR are parsed to yield a few keywords, on which a Web searching is executed to retrieve relevant and semantically informative articles from World Wide Web (WWW). The retrieved articles are then utilized to construct textual feature vectors and perform text categorization on behalf of commercials. For the second issue, a topic-wise document corpus is constructed from the public corpora like Reuters-21578 or from the articles manually collected from WWW for the training of text categorizers. Experimental results have shown that the proposed approach alleviates the negative effects from weak ASR/OCR performance and yield a promising classification accuracy of 80.9%

Precise automated video search is gaining in importance as the amount of multimedia information is increasing at exponential rates. One of the drawbacks that make video retrieval difficult is the lack of available semantics. In this paper, we propose to supplement the semantic knowledge for retrieval by providing useful semantic clusters derived from event entities present in the news video. These entities include the output from keywords derived from the automated speech recognition (ASR) and event-related High-level Features (HLF) extracted from the news video at the pseudo story level. Fuzzy clustering is then carried out to group similar stories together to form semantic clusters. The retrieval system utilizes these clusters to refine the re-ranking process in the Pseudo Relevance Feedback (PRF) step. Initial experiments performed on video search task using the TRECVID 2005 dataset show that the proposed approach can improve the search performance significantly.

In this paper, we describe a configurable content-based MPEG video authentication scheme, which is robust to typical video transcoding approaches, namely frame resizing, frame dropping and requantization. By exploiting the synergy between cryptographic signature, forward error correction (FEC) and digital watermarking, the generated content-based message authentication code (MAC or keyed crypto hash) is embedded back into the video to reduce the transmission cost. The proposed scheme is secure against malicious attacks such as video frame insertion and alteration. System robustness and security are balanced in a configurable way (i.e., more robust the system is, less secure the system will be). Compressed-domain process makes the scheme computationally efficient. Furthermore, the proposed scheme is compliant with state-of-the-art public key infrastructure. Experimental results demonstrate the validity of the proposed scheme

Motion characterization plays a critical role in video indexing. An effective way of characterizing camera motion facilitates the video representation, indexing and retrieval tasks. This paper describes a novel nonparametric motion representation to achieve an effective and robust recognition of parts of the video in which camera is static, or panning, or tilting, or zooming, etc. This representation employs the mean shift filtering and the vector histograms to produce a compact description of a motion field. The basic idea is to perform spatio-temporal mode-seeking in the motion feature space and use the histograms-based spatial distributions of dominant motion modes to represent a motion field. Unlike most existing approaches, which focus on the estimation of a parametric motion model from a dense optical flow field (OFF) or a block matching-based motion vector field (MVF), the proposed method combines the motion representation and machine learning techniques (e.g., support vector machines) to perform camera motion analysis from the classification point of view. The main motivation lies in the impossibility of uniformly securing a proper parametric assumption in a wide range of video scenarios. The diverse camera shot sizes and frequent occurrences of bad OFF/MVF necessitates a learning mechanism, which can not only capture the domain-independent parametric constraints, but also acquire the domain-dependent knowledge to tolerate the influence of bad OFF/MVF. In order to improve performance, we can use this learning-based method to train enhanced classifiers aiming at a certain context (i.e., shot size, neighbor OFF/MVFs, and video genre). Other visual cues (e.g., dominant color) can also be incorporated for further motion analysis. Our main aim is to use a generic feature space analysis method to explore a flexible OFF/MVF representation in a nonparametric technique, which could be fed into a learning framework to robustly capture the global motion by incorporating the context information. Results on videos with various types of content (23 191 MVFs culled from MPEG-7 dataset, and 20 000 MVFs culled from broadcast tennis, soccer, and basketball videos) are reported to validate the proposed approach.

This paper presents a novel trajectory-based detection and tracking algorithm for locating the ball in broadcast soccer video (BSV). The problem of ball detection and tracking in BSV is well known to be very challenging because of the wide variation in the appearance of the ball over frames. Direct detection algorithms do not work well because the image of the ball may be distorted due to the high speed of the ball, occlusion, or merging with other objects in the frame. To overcome these challenges, we propose a two-phase trajectory-based algorithm in which we first generate a set of ball-candidates for each frame, and then use them to compute the set of ball trajectories. Informally, the two key ideas behind our strategy are 1) while it is very challenging to achieve high accuracy in locating the precise location of the ball, it is relatively easy to achieve very high accuracy in locating the ball among a set of ball-like candidates and 2) it is much better to study the trajectory information of the ball since the ball is the "most active" object in the BSV. Once the ball trajectories are computed, the ball locations can be reliably recovered from them. One important advantage of our algorithm is that it is able to reliably detect partially occluded or merged balls in the sequence. Two videos from the 2002 FIFA World Cup were used to evaluate our algorithm. It achieves a high accuracy of about 81% for ball location

In this article, we propose a novel approach for automatic music video summarization. The proposed summarization scheme is different from the current methods used for video summarization. The music video is separated into the music track and video track. For the music track, a music summary is created by analyzing the music content using music features, an adaptive clustering algorithm, and music domain knowledge. Then, shots in the video track are detected and clustered. Finally, the music video summary is created by aligning the music summary and clustered video shots. Subjective studies by experienced users have been conducted to evaluate the quality of music summaries and effectiveness of the proposed summarization approach. Experiments are performed on different genres of music videos and comparisons are made with the summaries generated based on music track, video track, and manually. The evaluation results indicate that summaries generated using the proposed method are effective in helping realize users' expectations.

This paper proposes a content-based medical image retrieval (CBMIR) framework using dynamically optimized features from multiple regions of medical images. These regional features, including structural and statistical properties of color, texture and geometry, are extracted from multiple dominant regions segmented by applying Gaussian mixture modeling (GMM) and the expectation maximization (EM) algorithm to medical images. Over them, principal component analysis (PCA) is utilized to construct query templates and to reduce feature dimensions for representative feature optimization. Applying this method to the tasks of the medical imageCLEF 2004 we achieve better retrieval performance (MAP 0.4535) over the existing work on casImage of about 9000 images.

The automatic extraction of sports video highlights is a typical kind of personalized media production process. Many ways have been studied from the viewpoints of low-level audio/visual processing (e. g. detection of excited commentator speech), event detection (e. g. goal detection), etc. However, the subjectivity of highlights is an unavoidable bottleneck. The replay scene is an effective clue for highlights in broad-cast sports video due to the incorporation of video production knowledge. Most related work deals with the replay detection and/or a simple composition of all detected replays to generate highlights. Different from previous work, our work considers different flavors of different people in terms of highlight content or type through replay scenes classification. The main contributions include: 1) proposing a multi-modal (visual+ textual) approach for refined replay classification; 2) employing the sources of Broadcast Web Text (BWT) to facilitate replay content analysis. An overall accuracy of 79.9% has been achieved on seven soccer matches over seven replay categories

The development of mid-level concepts helps to bridge the gap between low-level feature and high-level semantics in video analysis. Most existing work combines the customized mid-level concepts and statistical models to detect particular events. Based on broadcast sports video production knowledge, we extend our previous work to present a unified framework for mid-level concept generation in this paper. A video segment is characterized via three essential aspects: camera shot size, an object appearing in a scene, and video production technology. These three aspects clearly summarize the primary concerns in terms of a generic concept generation. Within this framework, we can flexibly and clearly define meaningful mid-level concepts towards comprehensive video content analysis, such as replay classification and the detection of events (e. g. goal, shoot, attack, foul, offside, and out of bound, etc.).

Periodicity is useful for compact representation of periodic motion and a reasonable selection of a proper temporal scale for periodic motion analysis. In this paper, we concern the periodicity detection of local motion within an interesting region and present an approach to automatically detect the motion periodicity inherent to local motion under complex condition. The task is challenging as local motion is usually buried in clutters with global motion and noises. Most exist ing methods have assumed a static camera and a labeled moving object region. We instead apply robust local motion estimation and an object localization method to extract the object motion. The object motion is characterized by the confidence based motion probability map and the motion vectors obtained by global motion compensation. The autocorrelation series of motion energy is then carried out to locate local maximum points. With the set of indices of local maximum points, we can estimate the basic periodicity through a least-square fitting. This method has been applied to swimming videos and got encouraging results.

In this paper, we propose a novel automatic approach for personalized music sports video generation. Two research challenges, semantic sports video content selection and automatic video composition, are addressed. For the first challenge, we propose to use multi-modal (audio, video and text) feature analysis and alignment to detect the semantic of events in sports video. For the second challenge, we propose video-centric and music-centric music video composition schemes to automatically generate personalized music sports video based on user's preference. The experimental results and user evaluations are promising and show that our system's generated music sports video is comparable to manually generated ones. The proposed approach greatly facilitates the automatic music sports video generation for both professionals and amateurs.

We propose a novel framework for content-based image retrieval with multiple parallel retrieval engines (MultiPRE) to achieve higher retrieval performance. Visual features, including both low-level features, such as color, texture and region features, and middle-level structure features, such as blob representation of objects are used to capture geometrical and statistical characteristics of images. Both clustering analysis and discrimination analysis are used as similarity measures in multiple retrieval engines, which are based on~principal component analysis (PCA) and support vector machines (SVM), respectively. Finally outputs of these engines are fused to determine ranking lists of retrieved images for given retrieval topics. The proposed framework has been evaluated based on the 26 image query topics over the CasImage database~with over 9000 medical images~used in ImageCLEF 2004, an international research effort for content-based image retrieval performance benchmark. Experiments show that the proposed framework achieved significantly better performance in terms of both the mean and the variance of average precision than the best run reported in ImageCLEF2004.

Identifying short repeated video clips, such as news program logo, station logo, TV commercials, etc., from broadcasting video databases or streams is important for video content indexing, personalization as well as monitoring. In this demo system we present the following two functions: 1) automatically identifying variable length unknown repeated clips and known reference clips from video collections or steams; 2) visualize temporal distribution of repeated short video clips and analyze video structure based them. Experiment has been conducted on 12 hour CNN and ABC news videos, and excellent results have been achieved in both short repeated video clip identification and news video structure analysis.

In this paper, two discriminative fusion schemes are proposed for automatic image annotation. One is the ensemble-pattern association based fusion and another is the model-based transformation. The fusion approaches are studied and evaluated in a unified framework for AIA based on the text representation of the image content and the MC MFoM learning. The schemes are flexible for fusing diverse visual features and multiple modalities. The discriminative learning can automatically weight the most important features for the classification. We evaluate the fusion schemes based on the Corel and TRECVID 2003 datasets. The experimental results clearly show that the proposed fusion schemes give a significant improvement in term of the mean of F1as well as the number of the detected concepts

Scale estimation is important in image and vision computing. We propose in this paper an invariant salient region selection and scale normalization method which is robust to rotation, scaling, translation and cropping. This new method is based on the first and second order invariant geometric moments calculated from an intensity difference map. The first-order moments are used to obtain invariant circular regions for different scale hypotheses, while a second-order moment is chosen as region descriptor to select the most salient scale. The image is normalized by scale of the selected salient region. Experiments demonstrate effectiveness of this method

The extensive amount of multimedia information available necessitates content-based video indexing and retrieval methods. Since humans tend to use high-level semantic concepts when querying and browsing multimedia databases, there is an increasing need for semantic video indexing and analysis. For this purpose, we present a unified framework for semantic shot classification in sports video, which has been widely studied due to tremendous commercial potentials. Unlike most existing approaches, which focus on clustering by aggregating shots or key-frames with similar low-level features, the proposed scheme employs supervised learning to perform a top-down video shot classification. Moreover, the supervised learning procedure is constructed on the basis of effective mid-level representations instead of exhaustive low-level features. This framework consists of three main steps: 1) identify video shot classes for each sport; 2) develop a common set of motion, color, shot length-related mid-level representations; and 3) supervised learning of the given sports video shots. It is observed that for each sport we can predefine a small number of semantic shot classes, about 5-10, which covers 90%-95% of broadcast sports video. We employ nonparametric feature space analysis to map low-level features to mid-level semantic video shot attributes such as dominant object (a player) motion, camera motion patterns, and court shape, etc. Based on the fusion of those mid-level shot attributes, we classify video shots into the predefined shot classes, each of which has clear semantic meanings. With this framework we have achieved good classification accuracy of 85%-95% on the game videos of five typical ball type sports (i.e., tennis, basketball, volleyball, soccer, and table tennis) with over 5500 shots of about 8 h. With correctly classified sports video shots, further structural and temporal analysis, such as event detection, highlight extraction, video skimming, and table of content, will be greatly facilitated.

This letter describes a high-performance face recognition system by combining two recently proposed neural network models, namely Gabor wavelet network (GWN) and kernel associative memory (KAM), into a unified structure called Gabor wavelet associative memory (GWAM). GWAM has superior representation capability inherited from GWN and consequently demonstrates a much better recognition performance than KAM. Extensive experiments have been conducted to evaluate a GWAM-based recognition scheme using three popular face databases, i.e., FERET database, Olivetti-Oracle Research Lab (ORL) database and AR face database. The experimental results consistently show our scheme's superiority and demonstrate its very high-performance comparing favorably to some recent face recognition methods, achieving 99.3% and 100% accuracy, respectively, on the former two databases, exhibiting very robust performance on the last database against varying illumination conditions.

Effective and efficient representation of the low-level features of groups of frames or shots is an important yet challenging task for video analysis and retrieval. Key frame-based representation is limited by the difficulties in shot boundary detection of gradual transitions, and the variety of methods of key frame extraction. In this paper, we employ the mean shift-based mode seeking function to develop a new approach for compact representation of the video segment. The proposed video representation is motivated by recognizing that, on the global level, humans perceive images only as a combination of the few most prominent colors. We exploit the spatiotemporal mode seeking in feature space to simulate the "subjectivity" of human decisions in video segment retrieval and identification. The effectiveness of the video representation and matching scheme is shown by initial experiments on replay detection in broadcast sports videos.

The paper presents a generic, mid-level representation for efficient semantic video analysis, which adopts a frame-by-frame scheme using P-frames rather than shot-based schemes. Each P-frame is partitioned into an m/spl times/n grid (row by column), and each cell is called a 'block'. The representation can bridge the semantic gap and build an intermediate description of video features across frames and blocks. Soccer video is used to showcase the potential of the framework for real video processing. Experiments with tennis video and news video have also been conducted. Results demonstrate the excellent performance of the framework in semantic analysis and also indicate its further potential for automatic video analysis.

In this paper a novel sports news video shot classification method has been proposed. First two features based on motion and color are constructed and extracted from video shots: play field color ratio for specific types of sports, background motion and consistency ratio, then they are combined to generate an 11-dimension shot feature to feed into a C4.5 decision tree for shot classification. Based on our video data sets-the sports news video from the CNN Headline News video used in the TRECVID 2003, 7 predefined video shot classes were defined: 4 types of sports field video (basketball, baseball, ice hockey and golf) and sports news lead-in/lead-out, text and others. Sports news video segments from 15 half-hour CNN News video were used for the training and testing. A performance of average precision and recall 88%, 82% has been achieved, respectively. The proposed method can be further developed and used to search news video for individual sports news and sports highlights.

A multi-modal two-level framework for news story segmentation was proposed in Chaisorn et al. (2002). This paper presents our extended work scaled to cope with a large news video corpus used in TRECVID 2003 evaluation. We divided our system into two levels: the shot level that classifies input video shots into one of the predefined categories using a hybrid of heuristic and learning based approaches; and story level that performs story segmentation using the HMM framework based on the output of shot level and other temporal features. A heuristic rule-based technique is then employed to classify each detected story into "news" or "miscellaneous". We evaluated our system on over 120 hours of news video and showed that our system could achieve an accuracy of more than 77%. Our system came first in the TRECVID 2003 story segmentation task

A robust method for multi-level video representation based on the mean shift analysis (MSA) of low-level visual features is proposed in this paper. By tuning the bandwidth of MSA, video representation from the coarse level to the fine level can be achieved. This representation form provides a flexible scheme for content-based video analysis such as summarization, classification, and retrieval. Compared with the conventional k-means or fuzzy c-means algorithms, our method can adjust the resolution of representation in a more straightforward way, and is more robust since it does not need to initialize the cluster centers

Sports video highlight detection is a popular topic. A multi-layer sport event detection framework is described. In the mid-level of this framework, visual and audio keywords are created from low-level features and the original video is converted into a keyword sequence. In the high-level, the temporal pattern of keyword sequences is analyzed by an HMM classifier. The creation of visual and audio keywords can help to bridge the gap between low-level features and high-level semantics. The use of the HMM classifier can automatically find the temporal change character of the event instead of rule based heuristic modeling to map certain keyword sequences into events. Experiments using our model on soccer games produced some promising results.

Automatic music summarization is very useful for music indexing, content-based music retrieval and on-line music distribution, but it is a challenge to extract automatically the most common and salient themes from unstructured raw music data. We propose an effective approach to summarize music content automatically. First, a number of features are extracted to characterize the music content. Based on the extracted features, an adaptive clustering algorithm is then applied to structure the music content. Finally, the music summary is created in terms of the clustering results and domain-related music knowledge. A user study is conducted to evaluate the quality of summarization. The experiments on different genres of music illustrate the results of summarization are significant and effective to actual expectation.

We present a two dimensional timeline to replace the conventional 1D timeline used in video annotation systems. The extra dimension is augmented to represent the text annotations from user input and link them to the corresponding video time. A histogram of user annotations is developed over time. Not only does it highlight the history of prior user interaction with the video annotation system, it also gives a bird's eye view of how annotations are distributed along the video timeline. Users can then quickly find the annotation hotspots. Using this unique 2D timeline, we have developed a video annotation system prototype, Conversant Media (CM). Experimental trials conducted at two educational institutes on presentation skill training yield encouraging feedback, one of which concluded that CM can become a potential alternative to classroom visits for trainee teachers

This work presents a robust Hough-based algorithm for partial slightly oblique ellipse detection in broadcast soccer video. The successful identification of the ellipses significantly facilitate soccer video analysis. The existing standard and various modified ellipse Hough transforms measure a cell in the Hough space as though the ellipse defined by the cell were a complete ellipse. Hence, they are not robust when they are applied to detect the partial ellipses appearing in broadcast soccer video. This paper proposes a new measure function that is able to fairly measure whole and partial ellipses. With the improved measure function, we propose an algorithm to detect the partial ellipses in broadcast soccer video. The proposed algorithm first estimates the target ellipse by using the symmetry of the ellipse and the domain knowledge of soccer video. Then for each estimated ellipse, the algorithm searches around the estimated ellipses to find the ellipse with the highest measured value. Our algorithm is efficient and memory-small, i.e. it overcomes two main problems of the standard ellipse Hough transform. More importantly, the proposed algorithm is much more robust than the existing ellipse Hough transforms. Experimental results show that the proposed algorithm achieves above 96% recall and 100% precision. Our algorithm may be the first one that is able to detect ellipses from commercial video in pseudo real-time.

We propose a statistical approach to estimate object motion with MPEG motion vectors. A model with two normal distribution terms is applied to represent the simplified object motion. One term models the noise embedded in the motion vector field produced at the encoding stage and the other term models the randomness of the true object motion. Experiments with vehicle motion estimation from MPEG traffic video are used to evaluate the proposed algorithm. The influence of time window, frame size and reference frame distance are investigated. The vehicle speeds can be estimated with a high accuracy of up to 85-92%.

Motion information is a powerful cue for visual perception. In the context of video indexing and retrieval, motion content serves as a useful source for compact video representation. There has been a lot of literature about parametric motion models. However, it is hard to secure a proper parametric assumption in a wide range of video scenarios. Diverse camera shots and frequent occurrences of bad optical flow estimation motivate us to develop nonparametric motion models. In this paper, we employ the mean shift procedure to propose a novel nonparametric motion representation. With this compact representation, various motion characterization tasks can be achieved by machine learning. Such a learning mechanism can not only capture the domain-independent parametric constraints, but also acquire the domain-dependent knowledge to tolerate the influence of bad dense optical flow vectors or block-based MPEG motion vector fields (MVF). The proposed nonparametric motion model has been applied to camera motion pattern classification on 23191 MVF extracted from MPEG-7 dataset.

Motion information is a powerful cue for visual perception. In the context of video indexing and retrieval, motion content serves as a useful source for compact video representation. There has been a lot of literature about parametric motion models. However, it is hard to secure a proper parametric assumption in a wide range of video scenarios. Diverse camera shots and frequent occurrences of improper optical flow estimation or block matching motivate us to develop nonparametric motion models. In this demonstration, we present a novel nonparametric motion model. The unique features mainly include: 1) Instead of computationally expensive and vulnerable parametric regression our proposed model bases the motion characterization on the classification of motion patterns; 2) we employ machine learning to capture the knowledge of recognizing camera motion patterns from bad motion vector fields (MVF); and 3) with the mean shift filtering our proposed motion representation elegantly incorporates the spatial-range information for noise removal and discontinuity preserving smoothing of MVF. Promising results have been achieved on two tasks: 1) camera motion pattern recognition on 23191 MVFs and 2) recognition of the intensity of motion activity on 622 video segments culled from the MPEG-7 dataset.

Content based retrieval of similar multimedia objects (e.g. images, text, and videos) is an important research issue in the field of multimedia database. In this demo, we present a fast and robust video clip searching system. This system consists of two major modules, namely, robust video representation and fast searching. Different from traditional key frame-based histogram methods, we employ the cumulative histogram to represent the ordinal features and color range features for a video segment. This representation provides a spatio-temporal description of the whole segment. Our experiment has shown it is effective for capturing the patterns of short video clips such as commercial, program lead in/out, flying logo in sports video, etc. In order to improve the performance in searching large video database, we introduce the index structure to deal with video search from the viewpoint of query processing (e.g. K-NN query, Range query, etc.) in high-dimensional spaces. Different query processing support different search tasks. In this demo, we employ the mrkd-tree index structure and the proposed video representation to fulfill fast and robust search of short video clips (i.e. news video lead-in/out, replay logo, commercial) in large video collections with the total length of 15 hours.

While most current approaches for sports video analysis are based on broadcast video, in this paper, we present a novel approach for highlight detection and automatic replay generation for soccer videos taken by the main camera. This research is important as current soccer highlight detection and replay generation from a live game is a labor-intensive process. A robust multi-level, multi-model event detection framework is proposed to detect the event and event boundaries from the video taken by the main camera. This framework explores the possible analysis cues, using a mid-level representation to bridge the gap between low-level features and high-level events. The event detection results and mid-level representation are used to generate replays which are automatically inserted into the video. Experimental results are promising and found to be comparable with those generated by broadcast professionals.

In this paper we propose a novel space-time color feature representation for video shot and apply it to content identification. In this representation the shot is cut into k equal size segments, and each segment is represented by a blending image formed through averaging the pixels' values of each frame in this segment along time direction. Each blending image is then divided into equal size blocks, and two color patterns named major and minor colors among mean R,G,B are extracted for each block. Hence each shot can be represented by a fixed-length string. The experiment shows this representation is not only robust to image quality reduction, frame size and frame rate change, but also to color distortion such as brightness/contrast adjustment. We also give a video similarity measure based on this color feature to identify shot chunks. We conducted experiment on 100 video clips, and quite low error rates can be achieved when identifying small size shot chunks with significant color distortion. From the experiment we believe that this color feature is a compact and robust representation for video content, and effective for content identification.

The ellipse Hough transform (EHT) is a widely-used technique. Most of the previous modifications to the standard EHT improved either the voting procedure that computes the absolute measure function (AMF) or the peak detection of the AMF. However, existing EHTs are not robust for detecting partial slightly-oblique ellipses. This paper presents a Robust and Accumulator-Free Ellipse Hough Transform (RAF-EHT), an improved EHT that is robust even for partial slightly-oblique ellipses. Our RAF-EHT is based on two main ideas, namely, (1) an improved measure function (IMF) for handling the partiality and the obliqueness of ellipses, (2) a new accumulator-free computation scheme for finding the top k peaks of the IMF, without complex peak detection. Experimental results show that the RAF-EHT is more robust than the existing EHTs in detecting the partial slightly-oblique ellipses. In addition, the RAF-EHT needs only a little memory because it is accumulator-free.

Content-based video analysis calls for efficient video representation. In this paper, a novel multi-level representation of video is proposed based on the principle components derived from low-level visual features. It can characterize the video content from the coarse level to the fine level according to its intrinsic structure. This representation form provides a flexible scheme for video content analysis such as summarization, classification, and retrieval. A newly proposed subspace method, kernel based PCA, is explored to achieve this conveniently. The application in keyframe extraction is investigated to demonstrate the benefits of this representation.

This paper addresses the problem of background modeling for foreground object detection in complex environments. A Bayesian framework that incorporates spectral, spatial, and temporal features to characterize the background appearance is proposed. Under this framework, the background is represented by the most significant and frequent features, i.e., the principal features, at each pixel. A Bayes decision rule is derived for background and foreground classification based on the statistics of principal features. Principal feature representation for both the static and dynamic background pixels is investigated. A novel learning method is proposed to adapt to both gradual and sudden "once-off" background changes. The convergence of the learning process is analyzed and a formula to select a proper learning rate is derived. Under the proposed framework, a novel algorithm for detecting foreground objects from complex environments is then established. It consists of change detection, change classification, foreground segmentation, and background maintenance. Experiments were conducted on image sequences containing targets of interest in a variety of environments, e.g., offices, public buildings, subway stations, campuses, parking lots, airports, and sidewalks. Good results of foreground detection were obtained. Quantitative evaluation and comparison with the existing method show that the proposed method provides much improved results.

This paper presents an object based watermarking solution for MPEG4 video authentication. The watermark is embedded in the discrete Fourier transform (DFT) domain before MPEG4 encoding. Groups of DFT coefficients in the low-middle frequency band are selected for watermarking. The coefficients in every group are divided into two sub-groups based on a pre-defined pattern, and the energy relationship between these two sub-groups is used to hide the watermark. The experimental results show that our algorithm is robust against MPEG4 compression as well as object based video manipulations such as rotation, scaling and inaccurate segmentation.

We propose an effective fusion scheme of visual and auditory modalities to detect events in sports video. The proposed scheme is built upon semantic shot classification, where we classify video shots into several major or interesting classes, each of which has clear semantic meanings. Among major shot classes we perform classification of the different auditory signal segments (i.e. silence, hitting ball, applause, commentator speech) with the goal of detecting events with strong semantic meaning. For instance, for tennis video, we have identified five interesting events: serve, reserve, ace, return, and score. Since we have developed a unified framework for semantic shot classification in sports videos and a set of audio mid-level representation with supervised learning methods, the proposed fusion scheme can be easily adapted to a new sports game. We are extending this fusion scheme to three additional typical sports videos: basketball, volleyball and soccer. Correctly detected sports video events will greatly facilitate further structural and temporal analysis, such as sports video skimming, table of content, etc.

Automatic musical genre classification is very useful for music indexing and retrieval. In this paper, an efficient and effective automatic musical genre classification approach is presented. A set of features is extracted and used to characterize music content. A multi-layer classifier based on support vector machines is applied to musical genre classification. Support vector machines are used to obtain the optimal class boundaries between different genres of music by learning from training data. Experimental results of multi-layer support vector machines illustrate good performance in musical genre classification and are more advantageous than traditional Euclidean distance based method and other statistic learning methods.

In this paper, we proposed a robust moving video object segmentation algorithm using features in the MPEG compressed domain. We first cluster the motion vectors and produce a motion mask. Then, a difference mask at 8 x 8 block size is extracted from the DC image by background subtraction method. Finally, the motion mask and the difference mask are combined conditionally to generate the final object mask based on a set of rules that are application specified and is obtained with learning or heuristic based methods. The experimental results show that this object segmentation scheme is more robust than those using DC image or motion vectors only.

With rapid advances in sensor, storage, processor, and communication technologies, consumers can now afford to create, store, process, and share large digital photo collections. With more and more digital photos accumulated, consumers need effective and efficient tools to organize and access photos in a semantically meaningful way without too much manual annotation effort. From user studies, we confirm that users prefer to organize and access photos along semantic axes such as event, people, time, and place. In this paper, we propose a computational learning framework to construct event models from sample photos with event labels given by a user and to compute relevance measures of unlabeled photos to the event models. We demonstrate event-based retrieval on 2400 genuine home photos using our proposed approach.

In this paper, we present a secure and robust content authentication scheme for scalable video streaming. In our authentication scheme we consider three common video transcoding methods as acceptable content manipulations, when the streaming bit-rate needs to be reduced, namely frame resizing, frame dropping and multi-cycle coding. By employing error correction coding (ECC) in different ways, the proposed scheme is insensitive to those incidental distortions introduced during the transcoding (i.e., robust) while is still sensitive to other intentional distortions such as frame alterations and insertion (i.e., secure). One key feature in our scheme is that it achieves an end-to-end authentication independent of transcoding infrastructure and obtains a good compromise between system robustness and security.

In this paper, we propose an effective fusion scheme of visual and auditory modalities to detect events in sports video. The proposed scheme is built upon semantic shot classification, where we classify video shots into several major or interesting classes, each of which has clear semantic meanings. Among major shot classes we perform classification of the different auditory signal segments (i.e. silence, hitting ball, applause, commentator speech) with the goal of detecting events with strong semantic meaning. For instance, for tennis video, we have identified five interesting events: serve, reserve, ace, return, and score. Since we have developed a unified framework for semantic shot classification in sports videos and a set of audio mid-level representation with supervised learning methods, the proposed fusion scheme can be easily adapted to a new sports game. We are extending this fusion scheme to three additional typical sports videos: basketball, volleyball and soccer. Correctly detected sports video events will greatly facilitate further structural and temporal analysis, such as sports video skimming, table of content, etc.

This paper presents a novel framework called audio keywords to assist event detection in soccer video. Audio keyword is a middle-level representation that can bridge the gap between low-level features and high-level semantics. Audio keywords are created from low-level audio features by using support vector machine learning. The created audio keywords can be used to detect semantic events in soccer video by applying a heuristic mapping. Experiments of audio keywords creation and event detection based on audio keywords have illustrated promising results. According to the experimental results, we believe that audio keyword is an effective representation that is able to achieve more intuitionistic result for event detection in sports video compared with the method of event detection directly based on low-level features.

Despite a lot of research efforts in sports video analysis, soccer video indexing remains a challenging task due to the lack of structure in a soccer game that could help in structure analysis. In particular, little work was done in detecting and tracking the ball whose trajectory could play a crucial role for detecting key events. We propose a novel framework for accurately detecting the ball for broadcast soccer video. Our framework combines both direct and indirect insights to identify the ball rather than conventional simple template matching methods. It has three key components. First we infer the ball size range from the player size. Next non-ball objects are removed to reduce the possible ball candidates. Last but not least, a Kalman filer-based procedure mines candidate trajectories in candidate feature images. Then, a procedure selects the reliable ball trajectories from them. The experimental results on two 1000-frame sequences confirm that the proposed framework is very effective and obtain a better result than existing methods.

It is challenging to detect and track the ball from the broadcast soccer video. The feature-based tracking method to judge if a sole object is a target are inadequate because the features of the balls change fast over frames and we cannot differ the ball from other objects by them. This paper proposes a new framework to find the ball position by creating and analyzing the trajectory. The ball trajectory is obtained from the candidate collection by use of the heuristic false candidate reduction, the Kalman filter-based trajectory mining, and the trajectory evaluation. The ball trajectory is extended via a localized Kalman filter-based model matching procedure. The experimental results on two consecutive 1000-frame sequences illustrate that the proposed framework is very effective and can obtain a very high accuracy that is much better than existing methods.

Sports video has been widely studied due to its tremendous commercial potentials. Despite encouraging results from various specific sports games, it is almost impossible to extend a system for a new sports game because they usually employ different sets of low-level features appropriate for the specific games and closely coupled with the use of game specific rules to detect events or highlights. There is a lack of internal representation and structure to be generic and applicable for many different sports. In this paper, we present a generic mid-level representation framework for semantic sports video analysis. The mid-level representation layer is introduced between the low-level audiovisual processing and high-level semantic analysis. It allows us to separate sports specific knowledge and rules from the low-level and mid-level feature extraction. This makes sports video analysis more efficient, effective, and less ad-hoc for various types of sports. To achieve robustness of the low-level feature analysis, a non-parametric clustering, mean shift procedure, has been successfully applied to both color and motion analysis. The proposed framework has been tested for five field-ball type sports covering duration of about 8 hours. Experiments have shown its robust performance in semantic analysis and event detection. We believe that the proposed mid-level representation framework can be used for event detection, highlight extraction, summarization and personalization of many types of sports video. Categories and Subject Descriptors H.3.1 [Content Analysis and Indexing]: abstracting methods, indexing methods.

Color is very useful in locating and recognizing objects that occur in artificial environments. The color histogram has shown its efficiency and advantages as a general tool for various applications, such as content-based image retrieval and video browsing, object indexing and location, and video segmentation. However, due to the lack of any spatial and context information, the histogram is not robust and effective for color characterization (e.g. dominant color) in large video databases. In this paper, we propose a nonparametric color characterization model using mean shift procedure, with an emphasis on spatio-temporal consistency. Experimental results suggest that the color characterization model is much more effective for video indexing and browsing, particularly in the domain of structured video (e.g. sports video).

This paper proposes a novel method for detection and segmentation of foreground objects from a video which contains both stationary and moving background objects and undergoes both gradual and sudden "once-off" changes. A Bayes decision rule for classification of background and foreground from selected feature vectors is formulated. Under this rule, different types of background objects will be classified from foreground objects by choosing a proper feature vector. The stationary background object is described by the color feature, and the moving background object is represented by the color co-occurrence feature. Foreground objects are extracted by fusing the classification results from both stationary and moving pixels. Learning strategies for the gradual and sudden "once-off" background changes are proposed to adapt to various changes in background through the video. The convergence of the learning process is proved and a formula to select a proper learning rate is also derived. Experiments have shown promising results in extracting foreground objects from many complex backgrounds including wavering tree branches, flickering screens and water surfaces, moving escalators, opening and closing doors, switching lights and shadows of moving objects.

This paper first presents an improved trajectory-based algorithm for automatically detecting and tracking the ball in broadcast soccer video. Unlike the object-based algorithms, our algorithm does not evaluate whether a sole object is a ball. Instead, it evaluates whether a candidate trajectory, which is generated from the candidate feature image by a candidate verification procedure based on Kalman filter,, which is generated from the candidate feature image by a candidate verification procedure based on Kalman filter, is a ball trajectory. Secondly, a new approach for automatically analyzing broadcast soccer video is proposed, which is based on the ball trajectory. The algorithms in this approach not only improve play-break analysis and high-level semantic event detection, but also detect the basic actions and analyze team ball possession, which may not be analyzed based only on the low-level feature. Moreover, experimental results show that our ball detection and tracking algorithm can achieve above 96% accuracy for the video segments with the soccer field. Compared with the existing methods, a higher accuracy is achieved on goal detection and play-break segmentation. To the best of our knowledge, we present the first solution in detecting the basic actions such as touching and passing, and analyzing the team ball possession in broadcast soccer video.

This paper proposes a novel method for tracking persons based on the principal colors of human objects. First, an efficient human object representation method, principal color representation (PCR), is proposed. Asymmetric similarity measures are then proposed based on the principal color representation. These asymmetric similarity measures could be used to evaluate the matching between two individuals as well as visual evident of an individual in a group. An efficient algorithm for tracking persons as individuals or in groups is then described. The method has been tested using image sequences containing multiple moving persons frequently gathering and separating. Our test results have shown that proposed method has successfully tracked both persons as individuals or in groups, and is robust to illumination changes.

In this paper, we present an algorithm for automatic music summarization. Spectral power, amplitude envelope and mel-cepstral coefficients are calculated as features to characterize the segmented frames of the music. Based on calculated features, a clustering technique is applied to group segmented frames into different clusters to structure the music content. Finally, the music summary is generated based on the clustering results and domain-specific music knowledge. The proposed algorithm is tested on different genres of music samples and the results of summarization are effective to actual expectation after listening evaluation by experienced users.

In this demonstration, we present a system to analyze the clear degree of faces present in MPEG compressed video of Head-and-Shoulders style. The proposed system consists of three hierarchical modules: low-level features extraction, robust face tracking, and clear faces selection. We have integrated the core algorithm into an Automated Transaction Service (ATS) surveillance system. The Incremental Focus of Attention (IFA) architecture is taken to combine pixel domain processing with compressed domain processing --- thus, implemented system exhibits computational efficiency and tolerance to very cluttered scenes. The proposed system has successively detected segments with clear frontal faces from more than 20 Automated Teller Machine (ATM) testing clips in MPEG format, each of which consists of 1~3 transactions. In addition, the proposed scheme implies some potential video mining applications, such as automatic checking to verify entry authorization, retrieval of suspicious activities in prerecorded video surveillance sequences.

In this demonstration, we present a unified framework for semantic shot classification in sports videos. Unlike previous approaches, which focus on clustering by aggregating shots with similar low-level features, the proposed scheme makes use of domain knowledge of specific sport to perform a top-down video shot classification, including identification of video shots classes for each sport, and supervised learning and classification of given sports video with low-level and middle-level features extracted from the sports video. It's observed that for each sport we can predefine a small number of semantic shot classes, 5--10, which cover 90 to 95 % of sports broadcasting video. With supervised learning method, we can map the low-level features to middle-level semantic video shot attributes such as dominant object motion (a player), camera motion patterns, and court shape, etc. On the basis of the appropriate fusion of those middle-level shot attributes, we classify video shots into the predefined video shot classes, each of which has a clear semantic meaning. The proposed method has been tested over 3 types of sports videos: tennis, basketball, and soccer. Good classification results ranging from 80~95% have been achieved. The proposed framework provides a generic solution for sports video semantic shot classification, which can be adapted to a new sport type easily. With correctly classified sports video shots further structural and temporal analysis will be greatly facilitated.

This paper proposes a novel method for detecting foreground objects in nonstationary complex environments containing moving background objects. We derive a Bayes decision rule for classification of background and foreground changes based on inter-frame color co-occurrence statistics. An approach to store and fast retrieve color co-occurrence statistics is also established In the proposed method, foreground objects are detected in two steps. First, both foreground and background changes are extracted using background subtraction and temporal differencing. The frequent background changes are then recognized using the Bayes decision rule based on the learned color co-occurrence statistics. Both short-term and longterm strategies to learn the frequent background changes are proposed Experiments have shown promising results in detecting foreground objects from video containing wavering tree branches and flickering screens/water surface. The proposed method has shown better performance as compared with two existing methods.