A novel stroke based sparse hybrid convolutional neural networks (CNNs) method is proposed for author classification of Chinese ink-wash paintings (IWPs). As Chinese IWPs usually have many authors in several art styles, this differs from real images or western paintings and has led to a big challenge. In our work, we classify Chinese IWPs of different artists by analyzing a set of automatically extracted brushstrokes. A sparse hybrid CNNs in a deep-learning framework is then proposed to extract brushstroke features to replace the commonly used handcrafted ones such as edge, color, intensity and texture. Using 120 IWPs from six famous artists, promising results have been shown in successfully classifying authors in comparison to two other state-of-the-art approaches.

We present an automatic statistical intensity-based approach to extract the 3D cerebrovascular structure from time-of flight (TOF) magnetic resonance angiography (MRA) data. We use the finite mixture model (FMM) to fit the intensity histogram of the brain image sequence, where the cerebral vascular structure is modeled by a Gaussian distribution function and the other low intensity tissues are modeled by Gaussian and Rayleigh distribution functions. To estimate the parameters of the FMM, we propose an improved particle swarm optimization (PSO) algorithm, which has a disturbing term in speeding updating the formula of PSO to ensure its convergence. We also use the ring shape topology of the particles neighborhood to improve the performance of the algorithm. Computational results on 34 test data show that the proposed method provides accurate segmentation, especially for those blood vessels of small sizes.

AdaBoost classifiers with Haar-like features are widely used for license plate (LP) localization. However, it normally requires high-dimensional Haar-like features which cause extremely high computational cost. In this paper, a rejection cascade was built for LP localization with reduced Haar-like features. We first introduced line segment features as pre-input of Haar-like features for AdaBoost to eliminate more than 70% of the background in an image. Line segment features, including density, directionality and regularity, were extracted from line segments, which were detected by applying Hough Transform on an edge image. Later, AdaBoost classifiers with Haar-like features were further applied to identify the exact location of license plates. Our method dramatically reduced the demanded dimensions of Haar-like features, therefore saved much time in AdaBoost training stage. By comparing our method with methods of only using Haar-like features and only using line segment features, experimental results demonstrated that our proposed method achieved the best detection rate with significantly reduced dimensions of Haar-like features.

Movie shot classification is vital but challenging task due to various movie genres, different movie shooting techniques and much more shot types than other video domain. Variety of shot types are used in movies in order to attract audiences attention and enhance their watching experience. In this pa per, we introduce context saliency to measure visual attention distributed in keyframes for movie shot classification. Different from traditional saliency maps, context saliency map is generated by removing redundancy from contrast saliency and incorporating geometry constrains. Context saliency is later combined with color and texture features to generate feature vectors. Support Vector Machine (SVM) is used to classify keyframes into pre-defined shot classes. Different from the existing works of either performing in a certain movie genre or classifying movie shot into limited directing semantic classes, the proposed method has three unique features: 1) context saliency significantly improves movie shot classification; 2) our method works for all movie genres; 3) our method deals with the most common types of video shots in movies. The experimental results indicate that the proposed method is effective and efficient for movie shot classification.

Close-up (CU) is a photographic technique which tightly frames a person or an object. In movies, it is applied to guide audience attention and to evoke audience emotion. In this paper, we detect face CU, object CU, and lean of movies, which are widely used to romance emotions. A lean consists of shots in a sequence, with a close-up shot as focus. A set of features are extracted by considering movie making techniques and human attention for CU detection. The features are average saliency, color entropy, color variance, face height, skin area, and texture scales. These features are tested through statistical hypothesis test to be significantly discriminating for CUs. Then, Support Vector Machine (SVM) is applied on these features to detect face CU and object CU. Based on the face CU and object CU detection result, lean is further detected by investigating the changing of the face/object size. Lean detection is of challenge due to the technique of montage. We solve this problem through color similarity estimation and SIFT point matching. Experimental results on four full length movies verify the effectiveness of the proposed method.

Packaging appearance is extremely important in cigarette manufacturing. Typically, there are two types of cigarette packaging defects: (1) cigarette laying defects such as incorrect cigarette numbers and irregular layout; (2) tin paper handle defects such as folded paper handles. In this paper, an automated vision-based defect inspection system is designed for cigarettes packaged in tin containers. The first type of defects is inspected by counting the number of cigarettes in a tin container. First k-means clustering is performed to segment cigarette regions. After noise filtering, valid cigarette regions are identified by estimating individual cigarette area using linear regression. The k clustering centers and area estimation function are learned off-line on training images. The second kind of defect is detected by checking the segmented paper handle region. Experimental results on 500 test images demonstrate the effectiveness of the proposed inspection system. The proposed method also contributes to the general detection and classification system such as identifying mitosis in early diagnosis of cervical cancer.

TV commercials on broadcast television stations are repeats with essentially no changes. They can vary from several seconds in length to several minutes. The problem of finding repeating unknown commercials is highly related to video processing techniques, such as video segmentation, indexing and retrieval. In this paper, we propose a recognition system which shows how repeating commercials can be identified in a TV stream using a novel scheme. Our architecture assumes no a prior knowledge of TV streams or commercials and doesn't require a pre-trained database. The experimental result on the live TV streams show that the proposed algorithms are effective in detecting repeats in digital television video stream.

Emotional factors directly reflect audiences' attention, evaluation and memory. Affective contents analysis not only create an index for users to access their interested movie segments, but also provide feasible entry for video highlights. Most of the work focus on emotion type detection. Besides emotion type, emotion intensity is also a significant clue for users to find their interested content. For some film genres (Horror, Action, etc), the segments with high emotion intensity have the most possibilities to be video highlights. In this paper, we propose a hierarchical structure for emotion categories and analyze emotion intensity and emotion type by using arousal and valence related features hierarchically. Firstly, High, Medium and Low are detected as emotion intensity levels by using fuzzy c-mean clustering on arousal features. Fuzzy clustering provides a mathematical model to represent vagueness, which is close to human perception. After that, valence related features are used to detect emotion types (Anger, Sad, Fear, Happy and Neutral). Considering video is continuous time series data and the occurrence of a certain emotion is affected by recent emotional history, Hidden Markov Models (HMMs) are used to capture the context information. Experimental results shows the movie segments with high emotion intensity cover over 80% of the movie highlights in Horror and Action movies and the hierarchical method outperforms the one-step method on emotion type detection. Meanwhile, it is flexible for user to pick up their favorite affective content by choosing both emotion intensity levels and emotion types.

Due to the wide viewer-ship and high commercial potentials, recently, sports video analysis attracts extensive research efforts. One of the main tasks in sports video analysis is to identify sports genres i.e. sports video categorization. Most of the existing work focus on mapping content-based features to sports genres by using supervised learning methods. Moreover, video data sets seeks efficient data reduction methods due to the large size and noisy data. It lacks comparison analysis on the implementation and performance of these methods. In this paper, the research is carried out by using four dominant machine learning algorithms, namely Decision Tree, Support Vector Machine, K Nearest Neighbor and Naive Bayesian, and comparing their performance on a high dimensional feature set which selected by some feature selection tools such as Correlation-based Feature Selection (CFS), Principal Components Analysis (PCA) and Relief. Experimental results shows that Support Vector Machine (SVM) and k-NN are not sensitive to reduction of training sets. Moreover, three different feature reduction methods perform very differently with respect to four different tools.

With the advance of digital video recording and playback systems, the request for efficiently managing recorded TV video programs is evident so that users can readily locate and browse their favorite programs. In this paper, we propose a multimodal scheme to segment and represent TV video streams. The scheme aims to recover the temporal and structural characteristics of TV programs with visual, auditory, and textual information. In terms of visual cues, we develop a novel concept named program-oriented informative images (POIM) to identify the candidate points correlated with the boundaries of individual programs. For audio cues, a multiscale Kullback-Leibler (K-L) distance is proposed to locate audio scene changes (ASC), and accordingly ASC is aligned with video scene changes to represent candidate boundaries of programs. In addition, latent semantic analysis (LSA) is adopted to calculate the textual content similarity (TCS) between shots to model the inter-program similarity and intra-program dissimilarity in terms of speech content. Finally, we fuse the multimodal features of POIM, ASC, and TCS to detect the boundaries of programs including individual commercials (spots). Towards effective program guide and attracting content browsing, we propose a multimodal representation of individual programs by using POIM images, key frames, and textual keywords in a summarization manner. Extensive experiments are carried out over an open benchmarking dataset TRECVID 2005 corpus and promising results have been achieved. Compared with the electronic program guide (EPG), our solution provides a more generic approach to determine the exact boundaries of diverse TV programs even including dramatic spots.

Sports video has attracted a global viewership. Research effort in this area has been focused on semantic event detection in sports video to facilitate accessing and browsing. Most of the event detection methods in sports video are based on visual features. However, being a significant component of sports video, audio may also play an important role in semantic event detection. In this paper, we have borrowed the concept of the “keyword” from the text mining domain to define a set of specific audio sounds. These specific audio sounds refer to a set of game-specific sounds with strong relationships to the actions of players, referees, commentators, and audience, which are the reference points for interesting sports events. Unlike low-level features, audio keywords can be considered as a mid-level representation, able to facilitate high-level analysis from the semantic concept point of view. Audio keywords are created from low-level audio features with learning by support vector machines. With the help of video shots, the created audio keywords can be used to detect semantic events in sports video by Hidden Markov Model (HMM) learning. Experiments on creating audio keywords and, subsequently, event detection based on audio keywords have been very encouraging. Based on the experimental results, we believe that the audio keyword is an effective representation that is able to achieve satisfying results for event detection in sports video. Application in three sports types demonstrates the practicality of the proposed method.

TV commercial video is a kind of informative medium. To fast and robustly index and retrieve commercial videos is of interest to commercial monitor, copyright protection, and commercial management, we propose a coarse-to-fine scheme to robustly retrieve commercial videos. Different from previous work using clip or key frames-based matching, our scheme has incorporated the commercial production knowledge to search the candidate commercial positions. Color and ordinal features are extracted for locating the exact commercial positions with dynamic time warping distance. Comparison experiments were carried out over TRECVID 2006 news videos and some videos from Chinese channels. Our scheme has achieved promising simulation results.

Scene is a series of semantic correlated video shots. An effective scene detection depends on domain knowledge more or less. Most existing approaches try to directly detect various scene changes by applying clustering or supervised learning methods to low level audiovisual features. However, robustly detecting diverse scene changes derived from complex semantic meanings is still a challenging problem. In this paper we are focused on the association of visual signal changes (e.g. cuts, fade-in, fade-out, etc.) and audio signal changes (e.g. speaker change, background music change, etc.) to propose a mid-level scene change representation, which is meant to locate candidate scene change points by characterizing temporally uncorrelated properties of audio and visual track in the case of scene change happening. By incorporating domain knowledge, enhanced features can be further extracted to complement this representation to bridge semantic gap towards scene change detection. We utilize a camera motion estimation algorithm to detect visual signal changes. Such visual change positions are selected as time-stamp points. An alignment is performed to search for candidate audio signal change positions by multi-scale Kullback-Leibler(K-L) distance computing. Both metric-based K-L distance approach and model-based HMM are applied to determine true audio signal changes. The associated visual and audio signal changes are considered as the mid-level scene change representation. This representation has been successfully applied to detect boundaries of individual commercial in TV broadcast stream with an accuracy of around 95%. Particularly the systematic alignment approach can be utilized in video summarization.

In recent times, most of the computer-vision assisted automatic camera control policies are based on human events, such as speaker position changes. In addition to these events, in this paper, we introduce a set of natural camera control and multimedia synchronization schemes based on individual object interaction. We present our methods in detail, including head-pose calculation and laser pointer guidance, which are used to estimate the region of interest (ROI) for both hand-held and object-at-distance. We explain, from our results, of how these set of approaches have achieved robustness, efficiency and unambiguous object interaction during real-time video shooting

Most broadcast stations rely on TV logos to claim video content ownership or visually distinguish the broadcast from the interrupting commercial block. Detecting and tracking a TV logo is of interest to TV commercial skipping applications and logo-based broadcasting surveillance (abnormal signal is accompanied by logo absence). Pixel-wise difference computing within predetermined logo regions cannot address semi-transparent TV logos well for the blending effects of a logo itself and inconstant background images. Edge-based template matching is weak for semi-transparent ones when incomplete edges appear. In this paper we present a more robust approach to detect and track TV logos in video streams on the basis of multispectral images gradient. Instead of single frame based detection, our approach makes use of the temporal correlation of multiple consecutive frames. Since it is difficult to manually delineate logos of irregular shape, an adaptive threshold is applied to the gradient image in subpixel space to extract the logo mask. TV logo tracking is finally carried out by matching the masked region with a known template. An extensive comparison experiment has shown our proposed algorithm outperforms traditional methods such as frame difference, single frame-based edge matching. Our experimental dataset comes from part of TRECVID2005 news corpus and several Chinese TV channels with challenging TV logos

In this paper, we propose an approach for TV commercial video classification by the categories of advertised products or services (e.g. automobiles, healthcare products, etc). Since automatic speech recognition (ASR) and optical character recognition (OCR) can deliver meaningful textual information related to products or services, TV commercial video classification is formulated as the problem of text categorization. However, there exist two challenges. Firstly, the background music of TV commercials makes ASR techniques yield erroneous and deficient output transcripts. Secondly, even if ASR and OCR could work perfectly, the limited textual information from TV commercials do not suffice to train a generic and non-overfitting text categorizer. For the first issue, our approach resorts to the external resources to expand deficient ASR and OCR transcripts. The output transcripts of ASR and OCR are parsed to yield a few keywords, on which a Web searching is executed to retrieve relevant and semantically informative articles from World Wide Web (WWW). The retrieved articles are then utilized to construct textual feature vectors and perform text categorization on behalf of commercials. For the second issue, a topic-wise document corpus is constructed from the public corpora like Reuters-21578 or from the articles manually collected from WWW for the training of text categorizers. Experimental results have shown that the proposed approach alleviates the negative effects from weak ASR/OCR performance and yield a promising classification accuracy of 80.9%

TV advertising is ubiquitous, perseverant, and economically vital. Millions of people's living and working habits are affected by TV commercials. In this paper, we present a multimodal ("visual + audio + text") commercial video digest scheme to segment individual commercials and carry out semantic content analysis within a detected commercial segment from TV streams.Two challenging issues are addressed. Firstly, we propose a multimodal approach to robustly detect the boundaries of individual commercials. Secondly, we attempt to classify a commercial with respect to advertised products/services. For the first, the boundary detection of individual commercials is reduced to the problem of binary classification of shot boundaries via the mid-level features derived from two concepts: Image Frames Marked with Product Information (FMPI) and Audio Scene Change Indicator (ASCI). Moreover, the accurate individual boundary enables us to perform commercial identification by clip matching via a spatial-temporal signature. For the second, commercial classification is formulated as the task of text categorization by expanding sparse texts from ASR/OCR with external knowledge. Our boundary detection has achieved a good result of F1 = 93.7% on the dataset comprising 499 individual commercials from TRECVID'05 video corpus. Commercial classification has obtained a promising accuracy of 80.9% on 141 distinct ones. Based on these achievements, various applications such as an intelligent digital TV set-top box can be accomplished to enhance the TV viewer's capabilities in monitoring and managing commercials from TV streams.

In this paper, we present an event-on-demand (EoD)video adaptation system. The proposed system supports users in deciding their events of interest and considers network conditions to adapt video source by event selection and frame dropping.Firstly, events are detected by audio/video analysis and annotated by the description schemes (DSs)provided by MPEG-7 Multimedia Description Schemes (MDSs). And then, to achieve a generic adaptation solution, the adaptation is developed following MPEG-21 Digital Item Adaptation (DIA)framework. We look at early release of the MPEG-21 Reference Software on XML generation and develop our own system for EoD video adaptation in three steps:1) the event information is parsed from MPEG-7 annotation XML file together with bitstream to generate generic Bitstream Syntax Description (gBSD). 2) Users' preference, Network Characteristic and Adaptation QoS (AQoS) are considered for making adaptation decision. 3) adaptation engine automatically parses adaptation decisions and gBSD to achieve adaptation.Unlike most existing adaptation work, the system adapts video of events with interest according to users' preference. Implementation following MPEG-7 and MPEG-21 standards provides a generic video adaptation solution. gBSD based adaptation avoids complex video computation. 30 students from various departments were invited to test the system and their responses has been positive.

Motion characterization plays a critical role in video indexing. An effective way of characterizing camera motion facilitates the video representation, indexing and retrieval tasks. This paper describes a novel nonparametric motion representation to achieve an effective and robust recognition of parts of the video in which camera is static, or panning, or tilting, or zooming, etc. This representation employs the mean shift filtering and the vector histograms to produce a compact description of a motion field. The basic idea is to perform spatio-temporal mode-seeking in the motion feature space and use the histograms-based spatial distributions of dominant motion modes to represent a motion field. Unlike most existing approaches, which focus on the estimation of a parametric motion model from a dense optical flow field (OFF) or a block matching-based motion vector field (MVF), the proposed method combines the motion representation and machine learning techniques (e.g., support vector machines) to perform camera motion analysis from the classification point of view. The main motivation lies in the impossibility of uniformly securing a proper parametric assumption in a wide range of video scenarios. The diverse camera shot sizes and frequent occurrences of bad OFF/MVF necessitates a learning mechanism, which can not only capture the domain-independent parametric constraints, but also acquire the domain-dependent knowledge to tolerate the influence of bad OFF/MVF. In order to improve performance, we can use this learning-based method to train enhanced classifiers aiming at a certain context (i.e., shot size, neighbor OFF/MVFs, and video genre). Other visual cues (e.g., dominant color) can also be incorporated for further motion analysis. Our main aim is to use a generic feature space analysis method to explore a flexible OFF/MVF representation in a nonparametric technique, which could be fed into a learning framework to robustly capture the global motion by incorporating the context information. Results on videos with various types of content (23 191 MVFs culled from MPEG-7 dataset, and 20 000 MVFs culled from broadcast tennis, soccer, and basketball videos) are reported to validate the proposed approach.

In this article, we propose a novel approach for automatic music video summarization. The proposed summarization scheme is different from the current methods used for video summarization. The music video is separated into the music track and video track. For the music track, a music summary is created by analyzing the music content using music features, an adaptive clustering algorithm, and music domain knowledge. Then, shots in the video track are detected and clustered. Finally, the music video summary is created by aligning the music summary and clustered video shots. Subjective studies by experienced users have been conducted to evaluate the quality of music summaries and effectiveness of the proposed summarization approach. Experiments are performed on different genres of music videos and comparisons are made with the summaries generated based on music track, video track, and manually. The evaluation results indicate that summaries generated using the proposed method are effective in helping realize users' expectations.

The automatic extraction of sports video highlights is a typical kind of personalized media production process. Many ways have been studied from the viewpoints of low-level audio/visual processing (e. g. detection of excited commentator speech), event detection (e. g. goal detection), etc. However, the subjectivity of highlights is an unavoidable bottleneck. The replay scene is an effective clue for highlights in broad-cast sports video due to the incorporation of video production knowledge. Most related work deals with the replay detection and/or a simple composition of all detected replays to generate highlights. Different from previous work, our work considers different flavors of different people in terms of highlight content or type through replay scenes classification. The main contributions include: 1) proposing a multi-modal (visual+ textual) approach for refined replay classification; 2) employing the sources of Broadcast Web Text (BWT) to facilitate replay content analysis. An overall accuracy of 79.9% has been achieved on seven soccer matches over seven replay categories

The development of mid-level concepts helps to bridge the gap between low-level feature and high-level semantics in video analysis. Most existing work combines the customized mid-level concepts and statistical models to detect particular events. Based on broadcast sports video production knowledge, we extend our previous work to present a unified framework for mid-level concept generation in this paper. A video segment is characterized via three essential aspects: camera shot size, an object appearing in a scene, and video production technology. These three aspects clearly summarize the primary concerns in terms of a generic concept generation. Within this framework, we can flexibly and clearly define meaningful mid-level concepts towards comprehensive video content analysis, such as replay classification and the detection of events (e. g. goal, shoot, attack, foul, offside, and out of bound, etc.).

Periodicity is useful for compact representation of periodic motion and a reasonable selection of a proper temporal scale for periodic motion analysis. In this paper, we concern the periodicity detection of local motion within an interesting region and present an approach to automatically detect the motion periodicity inherent to local motion under complex condition. The task is challenging as local motion is usually buried in clutters with global motion and noises. Most exist ing methods have assumed a static camera and a labeled moving object region. We instead apply robust local motion estimation and an object localization method to extract the object motion. The object motion is characterized by the confidence based motion probability map and the motion vectors obtained by global motion compensation. The autocorrelation series of motion energy is then carried out to locate local maximum points. With the set of indices of local maximum points, we can estimate the basic periodicity through a least-square fitting. This method has been applied to swimming videos and got encouraging results.

We study the problem of affective content analysis. In this paper, we think of affective contents as those video/audio segments, which may cause an audience's strong reactions or special emotional experiences, such as laughing or fear. Those emotional factors are related to the users' attention, evaluation, and memories of the content. The modeling of affective effects depends on the video genres. In this work, we focus on comedy and horror films to extract the affective content by detecting a set of so-called audio emotional events (AEE) such as laughing, horror sounds, etc. Those AEE can be modeled by various audio processing techniques, and they can directly reflect an audience's emotion. We use the AEE as a clue to locate corresponding video segments. Domain knowledge is more or less employed at this stage. Our experimental dataset consists of 40-minutes comedy video and 40-minutes horror film. An average recall and precision of above 90% is achieved. It is shown that, in addition to rich visual information, an appropriate usage of special audios is an effective way to assist affective content analysis.

This paper presents a content-adaptive system for streaming reconstructed soccer goal events over networks with bandwidth limited to 1.5Mbps or below. The reconstruction module analyzes a soccer video to produce corresponding panoramic field model with localized motion trajectories to provide tactic analysis for user's better comprehension of the goal events. The transmission module designs 3 schemes for different bandwidth conditions (150Kbps-1.5Mbps), (56-150Kbps), and (<56Kbps). Each scheme can choose dynamically the video content, viewing quality and where the reconstruction happens based on the network conditions and computational power of connected devices. We demonstrate the effectiveness and scalability of our system in experimental results of adaptive transmission two soccer goal events under three typical bandwidth conditions.

Detail-on-demand hypervideo will provide a powerful mechanism to allow viewers to see additional information of video segments through hyperlinks. A large number of tools are devoted to the identification of selectable video objects and the synchronizatio

We propose a novel framework that facilitates automatic editing and authoring of multimedia using static and moving cameras in a dynamic scene. The framework incorporates several video techniques such as object tracking using mean shift and object recognition using Scaled Invariant Feature Transform (SIFT). These techniques are linked together by a comprehensive yet simple-to-program script authoring mechanism based on video event detection. These combined features empower the system to play a virtual director role in live video stream editing and multimedia integration. The system requires minimum human intervention and can leverage production efficiency for both novice and professional users. The experimental results from our prototype system demonstrate that this framework is achievable using inexpensive hardware and standard video cameras. Our system provides comprehensive pre-production authoring capabilities that lend towards integration of video and heterogonous multimedia elements in realtime. We have found this framework to be useful in many applications such as live video streaming, distance education, live entertainment, sports coverage and personal video broadcasting.

Three-dimensional model acquisition of an object is essential in many multimedia applications. Constructing three-dimensional models of objects from two-dimensional images is an old problem in the area of computer vision. There are many publications and our approach is specifically designed for constructing the depth map of a human face on frontal view, based on the head movement in a monocular setting. Our approach does not require a predefined face model. In this paper, along with the front view image of the user, three additional images with various head movement are captured, and the head pose is then calculated. The depth map is calculated through a triangular mesh, which the nodes on the mesh are the feature points that we calculate the depth with. Through image registration process, the feature points on the front view image are mapped to the other three images. Based on the head pose and the newly mapped coordinate, we could calculate the depth of the feature point. The depth results calculated from each of the three images are combined together to find the final depth value. Our approach does not require additional hardware, or predefined models. The result is not as good as we wished, but suggests avenues for future.

This paper describes a new method for authoring video-based hypermedia. It defines an XML-based (Extensible Markup Language) data modeling language called HyperVideo Authoring Language (HyVAL) for constructing structured documents in which the composition of internal video objects (segments, scenes, shots, frames and visual objects in frames) and external media objects (text, audio, images, html pages, etc.) is specified. The model allows an author to interactively specify various internal objects and relationship among internal and external objects. Through the language, a flexible presentation of video-based hypermedia is achieved and comprehensive viewer interactions with video objects are carried out. The experiment of structuring video-based hypermedia using HyVAL in our authoring and presentation environment is also described in this paper.

The extensive amount of multimedia information available necessitates content-based video indexing and retrieval methods. Since humans tend to use high-level semantic concepts when querying and browsing multimedia databases, there is an increasing need for semantic video indexing and analysis. For this purpose, we present a unified framework for semantic shot classification in sports video, which has been widely studied due to tremendous commercial potentials. Unlike most existing approaches, which focus on clustering by aggregating shots or key-frames with similar low-level features, the proposed scheme employs supervised learning to perform a top-down video shot classification. Moreover, the supervised learning procedure is constructed on the basis of effective mid-level representations instead of exhaustive low-level features. This framework consists of three main steps: 1) identify video shot classes for each sport; 2) develop a common set of motion, color, shot length-related mid-level representations; and 3) supervised learning of the given sports video shots. It is observed that for each sport we can predefine a small number of semantic shot classes, about 5-10, which covers 90%-95% of broadcast sports video. We employ nonparametric feature space analysis to map low-level features to mid-level semantic video shot attributes such as dominant object (a player) motion, camera motion patterns, and court shape, etc. Based on the fusion of those mid-level shot attributes, we classify video shots into the predefined shot classes, each of which has clear semantic meanings. With this framework we have achieved good classification accuracy of 85%-95% on the game videos of five typical ball type sports (i.e., tennis, basketball, volleyball, soccer, and table tennis) with over 5500 shots of about 8 h. With correctly classified sports video shots, further structural and temporal analysis, such as event detection, highlight extraction, video skimming, and table of content, will be greatly facilitated.

Detecting meaningful visual entities (e.g. faces, sky, foliage, buildings, etc.) based on supervised pattern classifiers has become a trend in content-based image retrieval. However, a drawback of the supervised learning approach is the need for manually labeled regions as training samples. We propose a semi-supervised framework to discover local semantic patterns and generate their samples for training with minimal human intervention. Image classifiers are first trained on local image blocks from a small number of labeled images. Then, local semantic patterns are discovered from clustering the image blocks with high classification output. Training samples are induced from cluster memberships for support vector learning to form local semantic pattern detectors. During retrieval, similarities based on local class pattern indexes and discovered pattern indexes are combined to rank images. Query-by-example experiments on 2400 unconstrained consumer photos with 16 semantic queries show that the combined matching approach outperformed the fusion of color and texture features significantly in average precision by 37%.

Unlike professional or domain-specific images, home photos vary significantly. They pose great challenge for content-based image retrieval. In this paper, we propose a Bayesian formulation to unify both local and global content-based similarities for image matching and demonstrate its superior retrieval performance on 2400 genuine home photos. Our proposed framework uses support vector machines to extract and combine intraimage and interclass semantics. Support vector detectors are first trained on semantically meaningful regions and used to form detection image indexes. The indexes then serve as input for support vector learning of image classifiers to generate class-relative indexes. During image retrieval, similarities based on both detection-based and class-relative indexes are combined to rank images. Query-by-example experiments on 2400 home photos with 16 semantic queries show that the combined matching approach is better than matching with single index. It also outperformed the fusion of color and texture features by 55% in average precision.

The paper presents a generic, mid-level representation for efficient semantic video analysis, which adopts a frame-by-frame scheme using P-frames rather than shot-based schemes. Each P-frame is partitioned into an m/spl times/n grid (row by column), and each cell is called a 'block'. The representation can bridge the semantic gap and build an intermediate description of video features across frames and blocks. Soccer video is used to showcase the potential of the framework for real video processing. Experiments with tennis video and news video have also been conducted. Results demonstrate the excellent performance of the framework in semantic analysis and also indicate its further potential for automatic video analysis.

We propose an adaptive semantic image indexing framework based on the notion of semantic support regions. Semantic support regions are salient image regions that exhibit semantic meanings and that can be derived statistically to span a new indexing space. We perform the learning and indexing of semantic support regions for 2400 heterogeneous consumer photos using support vector machines. We demonstrate the uniqueness and effectiveness of query by spatial icons, supported by our semantic indexing framework, with 12 queries.

Pose determination of the human head is vital in many applications, such as face recognition, face re-orientation in videoconferencing, distance learning, etc. The paper suggests a simple and effective approach for pose determination based on head movement. Our approach is designed for video-based situations, i.e. it takes the advantage of the availability of the previous frame and the initial pose. It needs one feature point to determine the pitch angle, the yaw angle and the roll angle. The number of feature points to be tracked is significantly less than in many other approaches in this area. It is fast and efficient, and produces a reasonable accuracy.

The lack of eye contact in video conference degrades the user experience. This problem has been known and studied for many years. To correct the eye contact, it requires face-reorientation and eye gaze correction. Pose determination and viewpoint determination is a vital step prior re-orientating the face. This paper suggests a novel, simple and effective approach for the determination of pose and viewpoint based on head movement. Our approach is designed for the video-based situations, that it takes the advantage of the availability of a previous image, and initially with a known pose or viewpoint. It needs one feature point to determine the pitch angle and the yaw angle, and two feature points to determine the roll angle. The number of feature points to be tracked is significantly less than many other researches suggested in this area. Our approach has little overhead to the face re-orientation process. It is fast and efficient, and produces reasonable accuracy.

The lack of eye contact in video conference degrades the user experience. This problem has been known and studied for many years. There are hardware-based solutions to the eye gaze problem; however, these specialized systems are not generally accessible. This paper assumed the depth of the scene taken from the monocular camera is normal distributed on the XY plane on the real world, the plane parallel to the view plane. The three dimensional normal curve is initially estimated from the face model of the user. By performing rotation operation on the normal curve, the orientation of the face is rectified. The approach suggested in this paper is fast and effective. It has the advantages of 3D modelling, but could save the steps on complex registration, texture mapping and rendering.

The notion of event conveys rich semantics to consumers in their collection of photos. Our user studies confirm that consumers would like to organize and access their digital photo memories by events, complemented by other semantic axes such as people, time, and place. In this paper, we address photo event retrieval by means of semantic support regions and images. Semantic support regions are salient image regions that exhibit semantic meanings and that can be learned from sample images to span a new indexing space. Semantic support images are key photos derived statistically to model photo events. Retrieval of a photo event is performed using a winner-take-all approach to compute the relevance measure of a given photo to the event models. This novel approach for photo event retrieval is experimented on 2400 heterogeneous consumer photos with very promising results.

With rapid advances in sensor, storage, processor, and communication technologies, consumers can now afford to create, store, process, and share large digital photo collections. With more and more digital photos accumulated, consumers need effective and efficient tools to organize and access photos in a semantically meaningful way without too much manual annotation effort. From user studies, we confirm that users prefer to organize and access photos along semantic axes such as event, people, time, and place. As a matter of fact, research on content-based image retrieval in the last decade is yet to bridge the semantic gap between feature-based indexes computed automatically and human preferences on query and retrieval. In this paper, we attempt to address this semantic gap by focusing on the notion of "event" in home photos. First we propose event taxonomy for home photos. Next we propose a computational learning framework to construct event models from sample photos with event labels given by a user and to compute relevance measures of unlabeled photos to the event models, Last but not least, we demonstrate event-based retrieval on 2400 genuine home photos using our proposed approach.

This paper presents a new method called 'Score-Block operation', which is originally from the general image retrieval concept, with k nearest neighbour classifier for automated categorization of interstitial pattern in chest radiographs. Before this method is applied on a chest radiograph, the lung field is extracted from the image and image feature vectors for each region of interest are obtained using power spectrum and Quasi-Gabor filter. There are three problems to solve: padding of the border, quantification of the interstitial abnormalities of chest radiograph and ribs and vessel shadows of lung field. The Score-Block operation solves the first and second problems. The third problem is solved by two partition image databases (PIDB), which are normal partition image database and abnormal partition image database. PIDBs consist of all normal and abnormal partition images including other superimposed organs, so our system is not sensitive to the ribs and vessels shadow.

With the development of hardware and large capacity communication facilities, video has been more and more widely used in every field. Delivering video over the Internet has posted a big problem. Video files are extremely big and need hours or even tens of hours to download. Much research has been attempted over low-level structures such as coding, buffer, protocol, agent and server. These techniques are either not scalable or impractical. This paper presents a system designed to allow efficient retrieving, browsing and real-time playing of videos through the Internet using a web browser. Testing results on resource usage and system usability are presented.

We propose a novel image representation called visual keyword histogram (VKH) for content-based indexing and retrieval. Visual keywords are domain-relevant visual prototypes (e.g. faces, foliage, buildings etc) with both perceptual appearance and textual semantics. Collectively, VKHs axe computed over spatial tessellation to represent the distribution of visual keywords in various parts of an image. To construct a vocabulary of visual keywords, an incremental neural network is deployed to learn visual keywords from examples. This allows us to build domain-specific visual vocabularies rapidly and incrementally. Last but not least, we propose a new visual query language called Query by Spatial Icons (QBSI) that allows a user to specify a query in terms of "what" and "where". A visual query term constrains whether a visual keyword should be present and a query formals chains these terms into a disjunctive normal form via logical operators. We show our approach on real and complex home photos with very promising results.

Motion-based segmentation plays an important role in dynamic scene analysis of video sequences. We present a scheme for extracting moving objects. First, three different resolutions of the dense optical flow fields are calculated using a complex discrete wavelet transform. Surface fitting of all levels of these vectors is then performed over the affine parametric motion model. Next, the clustering by the competitive agglomeration algorithm is applied in the parameter space of the coarsest level. The results of this step are the optimum number of clusters and the center of each cluster. Using information from the previous level, the parameter spaces of the following levels are then segmented using the classical mixture model and the expectation-maximization algorithm. Finally, the individual moving object and background are represented in layers. Experimental results showing the significance of this proposed method are provided.

The nonlinear anisotropic diffusive process has shown the good property of eliminating noise while preserving the accuracy of edges and has been widely used in image processing. However, filtering depends on the threshold of the diffusion process, i.e., the cut-off contrast of edges. The threshold varies from image to image and even from region to region within an image. The problem compounds with intensity distortion and contrast variation. We have developed an adaptive diffusion scheme by applying the Central Limit Theorem to selecting the threshold. Gaussian distribution and Rayleigh distribution are used to estimate the distributions of visual objects in images. Regression under such distributions separates the distribution of the major object from other visual objects in a single-peak histogram. The separation helps to automatically determine the threshold. A fast algorithm is derived for the regression process. The method has been successfully used in filtering various medical images.

This paper presents a novel approach to stabilize the output of video camera installed on a moving vehicle in a rugged environment. A 2.5D interframe motion model is proposed so that the stabilization system can perform in situations where significant depth changes are present and the camera has both rotation and translation. Inertial motion filtering is proposed in order to eliminate the vibration of the video sequences with enhanced perceptual properties. The implementation of this new approach integrates four modules: pyramid-based motion detection, motion identification and 2.5D motion parameter estimation, inertial motion filtering, and affine-based motion compensation. The stabilization system can smooth unwanted vibrations or shakes of video sequences and achieve real-time speed. We test the system on IBM PC compatible machines and the experimental results show that our algorithm outperforms many algorithms which require parallel pipeline image processing machines.

Article An efficient nearest-neighbour search while varying Euclidean metrics Share on Authors: R. Kurniawati University of New South Wales, Sydney 2052, Australia University of New South Wales, Sydney 2052, AustraliaView Profile , J. S. Jin University of New South Wales, Sydney 2052, Australia University of New South Wales, Sydney 2052, AustraliaView Profile , J. A. Shepherd University of New South Wales, Sydney 2052, Australia University of New South Wales, Sydney 2052, AustraliaView Profile Authors Info & Claims MULTIMEDIA '98: Proceedings of the sixth ACM international conference on MultimediaSeptember 1998 Pages 411–418https://doi.org/10.1145/290747.290812Online:01 September 1998Publication History 2citation597DownloadsMetricsTotal Citations2Total Downloads597Last 12 Months3Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Article A signature for content-based image retrieval using a geometrical transform Share on Authors: H. Wang Department of Computer Science, The University of Sydney, NSW 2006, Australia Department of Computer Science, The University of Sydney, NSW 2006, AustraliaView Profile , F. Guo Department of Computer Science, The University of Sydney, NSW 2006, Australia Department of Computer Science, The University of Sydney, NSW 2006, AustraliaView Profile , D. D. Feng Department of Computer Science, The University of Sydney, NSW 2006, Australia and DoEE, Hong Kong Polytechnic University, Hong Kong Department of Computer Science, The University of Sydney, NSW 2006, Australia and DoEE, Hong Kong Polytechnic University, Hong KongView Profile , J. S. Jin School of Computer Science and Engineering, University of New South Wales, Sydney 2052, Australia School of Computer Science and Engineering, University of New South Wales, Sydney 2052, AustraliaView Profile Authors Info & Claims MULTIMEDIA '98: Proceedings of the sixth ACM international conference on MultimediaSeptember 1998 Pages 229–234https://doi.org/10.1145/290747.290775Online:01 September 1998Publication History 13citation675DownloadsMetricsTotal Citations13Total Downloads675Last 12 Months3Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

This paper describes a hierarchical matching method based on multichannel filtering. Zero-crossings extracted from a set of one-octave LoG filters provide features at different scales. A signature using optimal coding is proposed and has a higher entropy than Fourier descriptors. Voronoi diagram is used to guide matching from coarse to fine. The method has been successfully applied for matching electrophoretic images.