Motion analysis automatically captures, recognizes and predicts ongoing human activities, which can be widely applied to various useful domains such as security surveillance in public spaces, including shopping centers and airports. With the development of the technologies like 3D specialized markers, we could capture the moving signals from marker joints and create a huge set of 3D motion capture (MOCAP) data. We propose in this work a method to automatically extract the action features which can be used for action recognition. We create an depth architecture model by combining multilevel networks which can focus on the recognizing objects in detail. These networks can learn the extracted features and perform action recognition. This propose model not only can extract the semantic action features from 3D MOCAP data, but also can apply for the real-time action recognition.

In this paper, the Gabor filter is studied and further expanded for temporal facial expression analysis. Originally, the Gabor feature describes both spatial and frequency characteristics of 2D images. The prominent of the theorem has been validated in research communities for a decade due to its similarity to the human perception system. The performance of the filter in the existing research gives convincing results on recognizing the human emotions by using a still image. However, the previous research neglects the fact that the understanding of human facial expression of emotions is associated by the dynamic relation, which the motion of expression must be witnessed. Therefore, we propose the novel temporal features by deriving the dynamic of Gabor features in the temporal template representations. Then, we decompose the features onto discriminative subspace for estimating the emotion class.

Emergence of novel techniques devices e.g., MS Kinect, enables reliable extraction of human skeletons from action videos. Taking skeleton data as inputs, we propose an approach to extract the discriminative patterns for efficient human action recognition. Each action is considered to consist of a sequence of unit actions, each of which is represented by a pattern. Given a skeleton sequence, we first automatically extract the key-frames, and then categorize them into different patterns. We further use a statistical metric to evaluate the discriminative capability of patterns, and define them as local features for action recognition. Experimental results show that the extracted local descriptors could provide very high accuracy in the action recognition, which demonstrate the efficiency of our method in extracting discriminative unit actions.

We propose a hybrid personalized summarization framework that combines adaptive fast-forwarding and content truncation to generate comfortable and compact video summaries. We formulate video summarization as a discrete optimization problem, where the optimal summary is determined by adopting Lagrangian relaxation and convex-hull approximation to solve a resource allocation problem. To trade-off playback speed and perceptual comfort we consider information associated to the still content of the scene, which is essential to evaluate the relevance of a video, and information associated to the scene activity, which is more relevant for visual comfort. We perform clip-level fast-forwarding by selecting the playback speeds from discrete options, which naturally include content truncation as special case with infinite playback speed. We demonstrate the proposed summarization framework in two use cases, namely summarization of broadcasted soccer videos and surveillance videos. Objective and subjective experiments are performed to demonstrate the relevance and efficiency of the proposed method.

We propose a method for detecting group interactions for groups of varying number of objects. We model each object as a moving agent with a direction-aware interest map and group interactions as mutual interests between objects. After grouping objects into unit interactions individually in each frame, we solve the temporal association problem by tracking group interaction over consecutive frames. Optimal grouping is obtained by finding the maximum weight spanning tree of a directed graph formed by objects and their potential interactions. Experimental results show that our method obtained around 80% recalling rates on two publicly available datasets.

With the development of the technology like 3D specialized markers, we could capture the moving signals from marker joints and create a huge set of 3D action MoCap data. The more we understand the human action, the better we could apply it to applications like security, analysis of sports, game etc. In order to find the semantically representative features of human actions, we extract the sets of action characteristics which appear frequently in the database. We then propose an Apriori-like algorithm to automatically extract the common sets shared by different action classes. The extracted representative action characteristics are defined in the semantic level, so that it better describes the intrinsic differences between various actions. In our experiments, we show that the knowledge extracted by this method achieves high accuracy of over 80% in recognizing actions on both training and testing data.

A novel method is presented for trackingmultiple deformable objects in a multiview scenario. We avoid applying fixed constraints of object sizes for object isolation, but guide the splitting/merging of objects by grouping partial motion trajectories extracted from the occupancy status of objects in the homography ground-plane. We treat this clustering task as solving a rooted arborescence problem, and investigate the performance of the proposed method by experimental results.

When summarizing a video consisting of temporally continuous actions, e.g., surveillance videos or team-sport videos, fast-forwarding usually provides a better solution than content truncation to organize a semantically more complete and thus easily understandable story. Based on the resource allocation framework initially introduced in [1], we propose a fully automatic summarization system for broadcasted soccer videos, which supports adaptive fast-forwarding as well as content truncation. The proposed system involves two major contributions (i) an improved algorithm for refining the boundaries of far-view clips based on the extracted camera movement; (ii) a process to organize a summary by determining both the selected clips and their optimal fast-forwarding strategies, considering the scene-changing tempos of those clips and the given duration constraint. Experiments on real-life broad-casted soccer videos demonstrate the relevance and the efficiency of our proposed method.

Compared to conventional synthesis methods of free viewpoint images that use only multi-view images, our research synthesizes high quality free viewpoint images by using multi-view depth information as well as the images. By recovering high resolution and high precision 3D shapes from multi-view information, high quality free viewpoint images are synthesizable. Our research captures the scene by acquiring its multi-view depth appearance information, using a laser range finder. By performing camera parameter estimation, multi-view 3D shape integration and depth estimation, high quality free viewpoint images for any scene are synthesized. The performance of our method has been investigated by experimental results.

We propose a flexible framework to summarize team-sport videos that have been originally produced for broadcast purposes. The framework is able to integrate both the knowledge about displayed content (e.g., level of interest, type of view, and so on), and the individual (narrative) preferences of the user. It builds on the partition of the original video sequence into independent segments, and creates local stories by considering multiple ways to render each segment. We discuss how to segment videos automatically based on production principles, and design parametric functions to evaluate the benefit of various local stories from a segment. Summarization by selection of local stories is then regarded as a resource allocation problem, and Lagrangian relaxation is performed to find the optimum. We investigate the efficiency of our framework by summarizing soccer, basketball and volleyball videos in our experiments.

Democratic and personalized production of multimedia content is a challenge that content providers will have to face in the near future. In this paper, we address this challenge by building on computer vision tools to automate the collection and distribution of audiovisual content. Especially, we proposed a complete production process of personalized video summaries in a typical application scenario, where the sensor network for media acquisition is composed of multiple cameras, which, for example, cover a basketball field. Distributed analysis and interpretation of the scene are exploited to decide what to show or not to show about the event, so as to produce a video composed of a valuable subset of the streams provided by each individual camera. Interestingly, the selection of the streams subsets to forward to each user depends on his/her individual preferences, making the process adaptive and personalized. The process involves numerous integrated technologies and methodologies, including but not limited to automatic scene analysis, camera viewpoint selection, adaptive streaming, and generation of summaries through automatic organization of stories. The proposed technology provides practical solutions to a wide range of applications, such as personalized access to local sport events through a web portal, cost-effective and fully automated production of content dedicated to small-audience, or even automatic log in of annotations.

We propose a flexible framework for producing highly personalized basketball video summaries, by integrating contextural information, narrative user preferences on story pattern, and general production principles. Starting from the multiple streams captured by a distributed set of fixed cameras, we study the implementation of autonomous viewpoint determination and automatic temporal segment selection, and also discuss the production of visually comfortable output, by applying smoothing process to viewpoint selection and by defining efficient benefit functions to evaluate various summary organization. The efficiency of our framework is demonstrated by experimental results.

This paper describes two novel facial expression recognition methods which are robust for head rotation within a certain angle range between -30 degrees and +30 degrees. We had proposed Eigenspace Method based on Class features of object (EMC) and Multiple Discriminant Analysis (MDA) for facial expression recognition. Our new methods, pEMC (parametric Eigenspace Method based on Class Features) and pMDA (parametric Multiple Discriminant Analysis), are extensions of EMC and MDA by using the parametric eigenspace technique. The parametric technique finds the manifold vector for recognition of rotated objects. Since EMC and MDA have the higher class separation, our new methods have both characteristics of parametric eigenspace and high classification of facial expression. pEMC and pMDA provide more 20 degree correct recognition than EMC regardless of the head pose.

This paper presents a fully automatic system for soccer game summarization. The system takes audio-visual content as an input, and builds on the integration of two independent but complementary contributions (i) to identify crucial periods of the soccer game in a fully automatic way, and (ii) to summarize the soccer game as a function of individual narrative preferences of the user. The process involves both audio and video analysis, and handles the personalized summarization challenge as a resource allocation problem. Experiments on real-life broadcasted content demonstrate the relevance and the computational efficiency of our integrated approach.

We propose a flexible summarization framework for team-sport videos, which is able to integrate both the knowledge about displayed content (e.g. level of interest, type of view, etc.), and the individual (narrative) preferences of the user. Our framework builds on the partition of the original video sequence into independent segments, and create local stories by considering multiple ways to render each segment. We discuss how to segment videos based on production principles, and design the benefit function to evaluate various local stories from a segment. Summarization by selection of local stories is regarded as a resource allocation problem, and Lagrangian relaxation is performed to find the optimum. We use a soccer video to validate our framework in our experiments.