In this paper a view and scale insensitive action representation VSI-Surf is proposed. Scale invariant shape descriptor R-transform is used to extract compact 1D feature from view insensitive posture representation “Envelop shape” which uses only two orthogonal cameras without accurate calibration. Considering action is a posture sequence, to integrate temporal information, 1D posture feature is then extended in time dimension. Then we get an action representation insensitive to viewpoint and scale, which is called VSI-Surf. Actions recognition is processed in a hierarchical framework, in which body actions and gestures are recognized in different level. Encouraging recognition results have been demonstrated on the multi-view IXMAS action dataset.

Person localization is fundamental in human centered computing, since person should be localized before being actively serviced. This paper proposed a robust approach to localize person based on the geometric constraints in multi-camera environment. The proposed algorithm has several advantages: 1) no assumption on the positions and orientations of cameras except the cameras should have certain common field of view; 2) no assumption on the visibility of particular body part (e.g., feet), except a portion of person should be observed in at least two views; 3) reliability in terms of tolerating occlusion, body posture change and inaccurate motion detection. It can also provide error control and be further extended to measure person height. The efficacy of the approach is demonstrated on challenging real-world scenarios.

Traditional appearance-based head pose estimation methods use the holistic face appearance as the input and then employ subspace analysis methods to extract low-dimensional features for classification. However, the face appearance may be more related to the unique identity of an individual rather than head poses. In this paper, we presented a comparative study of two image representations which aim to specifically describe head pose variations. The histogram of oriented gradient (HOG) based method relies on the gradient orientation distribution. The GaFour method exploits asymmetry in the intensities of each row of the face image, using a Gabor filter and Fourier transform to represent the face images. We compare the two image representations combined with two linear subspace methods (PCA and LDA). Experiments on two public face databases (CMU-PIE and CAS-PEAL) show that both HOG+LDA and GaFour+LDA give good results and HOG+LDA provides the best performance with a lower feature dimension.

Elastic motion is a nonrigid motion constrained only by some degree of smoothness and continuity. Consequently, elastic motion estimation by explicit feature matching actually contains two correlated subproblems: shape registration and motion tracking, which account for spatial smoothness and temporal continuity, respectively. If we ignore their interrelationship, solving each of them alone will be rather challenging, especially when the cluttered features are involved. To integrate them into a probabilistic model, one straightforward approach is to draw the dependence between their hidden states. With regard to their separated states, there are, however, two different explanations of motion which are still made under the individual constraint of smoothness or continuity. Each one can be error-prone, and their coupling causes error propagation. Therefore, it is highly desirable to design a probabilistic model in which a unified state is shared by the two subproblems. This paper is intended to propose such a model, i.e., a Mixture of Transformed Hidden Markov Models (MTHMM), where a unique explanation of motion is made simultaneously under the spatiotemporal constraints. As a result, the MTHMM could find a coherent global interpretation of elastic motion from local cluttered edge features, and experiments show its robustness under ambiguities, data missing, and outliers.

Computer understanding of human actions and interactions is one of the key research issues in human computing. In this regard, context plays an essential role in semantic understanding of human behavioral and social signals from sensor data. This paper put forward an event-based dynamic context model to address the problems of context awareness in the analysis of group interaction scenarios. Event-driven multilevel dynamic Bayesian network is correspondingly proposed to detect multilevel events, which underlies the context awareness mechanism. Online analysis can be achieved, which is superior over previous works. Experiments in our smart meeting room demonstrate the effectiveness of our approach.

This paper proposes a novel Multi-layer Homography Algorithm for background modeling from a free-moving camera. Background is composed of many planes. Different planes satisfy with different homographies which can be found by our algorithm. Each pixel except for the moving pixel definitely belongs to some plane. Rectified by the corresponding homography, each static pixel in the shared view can find its match in the previous frame. Thus, frames can be rectified to a specific viewpoint for background modeling. Experiment shows it is effective. Our approach can be used in motion detection from a free-moving camera.

Computer understanding of human actions and interactions is one of the key research issues in human computing. In this regard, context plays an essential role in semantic understanding of human behavioral and social signals from sensor data. This paper put forward an event-based dynamic context model to address the problems of context awareness in the analysis of group interaction scenarios. Event-driven multilevel dynamic Bayesian network is correspondingly proposed to detect multilevel events, which underlies the context awareness mechanism. Online analysis can be achieved, which is superior over previous works. Experiments in our smart meeting room demonstrate the effectiveness of our approach.

Groupwise shape registration of raw edge sequence is addressed. Automatically extracted edge maps are treated as noised input shape of the deformable object and their registration are considered, results can be used to build statistical shape models without laborious manual labeling process. Dealing with raw edges poses several challenges, to fight against them a novel spatio-temporal generative model is proposed which joints shape registration and trajectory tracking. Mean shape, consistent correspondences among edge sequence and associated non-rigid transformations are jointly inferred under EM framework. Our algorithm is tested on real video sequences of a dancing ballerina, talking face, and walking person. Results achieved are interesting, promising, and prove the robustness of our method. Potential applications can be found in statistical shape analysis, action recognition, object tracking, etc.

For video summarization and retrieval, one of the important modules is to group temporal-spatial coherent shots into high-level semantic video clips namely scene segmentation. In this paper, we propose a novel scene segmentation and categorization approach using normalized graph cuts(NCuts). Starting from a set of shots, we first calculate shot similarity from shot key frames. Then by modeling scene segmentation as a graph partition problem where each node is a shot and the weight of edge represents the similarity between two shots, we employ NCuts to find the optimal scene segmentation and automatically decide the optimum scene number by Q function. To discover more useful information from scenes, we analyze the temporal layout patterns of shots, and automatically categorize scenes into two different types, i.e. parallel event scenes and serial event scenes. Extensive experiments are tested on movie, and TV series. The promising results demonstrate that the proposed NCuts based scene segmentation and categorization methods are effective in practice.

Unlike traditional 3D model based or image based animation methods, in this paper a novel approach is presented to generate both facial actions and head rotations for photo-realistic facial animation based on one frontal and one half-profile facial image taken with an uncalibrated camera. We represent faces with 2D wire-frame models and use MPEG4 FAPs to encode basic facial actions. Hierarchical Direct Appearance Model is employed for facial feature localization. 3D deformable model is applied for pose estimation. By affine projection 3D deformable model and facial actions are mapped to 2D facial models and actions at various head poses. Coarse 2D models are refined with extracted facial features by RBF interpolation. Pose-variable facial animation is generated by synthesizing facial actions on 2D models and morphing facial textures between frontal and halfprofile views. Experimental results demonstrate the effectiveness of our approach.

Color has been widely used in sports video analysis. Previous techniques, however require color models from prior information or user interaction, and do not address the problem of how to automatically form color models from a video in an arbitrary sports setting. In this paper, we propose an automatic technique for extracting color models of the playing surface and the team uniforms, which can be used in higher-level processes such as tracking and recognition. Unlike most previous methods, our approach is capable of handling multi-colored patterns like striped uniforms and playing fields. Multiple forms of color processing are used to analyze video frame content, which are then used iteratively to refine the color models. The results of our color modeling technique have been applied to shot classification, and experiments on videos of different sports have verified our approach.

This paper proposes an improved algorithm for ASM addressing at face alignment with pose variation conquering local obstruction. To providing good initialization of ASM for reliable searching starting-point and faster convergence, an affine transform insensitive initialization algorithm (ATIIA) is employed after several facial feature points are extracted by SDAM searching. Landmarks of ASM are sampled with special strategy to sufficiently describe gray-level information surrounding them. Special searching strategy is also provided for ASM so that problem of local obstruction can be solved. For multi-view face alignment, view-based ASMs are trained respectively for 5 face pose based on CMU-PIE database. The exciting experiment results are shown proving that this method is robust and not susceptible to various factors such as the scale of image, the rotation of head, the local obstruction and so on.

Multi-level annotation of images is a promising solution to enable more effective semantic image retrieval by using various keywords at different semantic levels. In this paper, we propose a multi-level approach to annotate the semantics of natural scenes by using both the dominant image components and the relevant semantic concepts. In contrast to the well-known image-based and region-based approaches, we use the salient objects as the dominant image components to achieve automatic image annotation at the content level. By using the salient objects for image content representation, a novel image classification technique is developed to achieve automatic image annotation at the concept level. To detect the salient objects automatically, a set of detection functions are learned from the labeled image regions by using Support Vector Machine (SVM) classifiers with an automatic scheme for searching the optimal model parameters. To generate the semantic concepts, finite mixture models are used to approximate the class distributions of the relevant salient objects. An adaptive EM algorithm has been proposed to determine the optimal model structure and model parameters simultaneously. We have also demonstrated that our algorithms are very effective to enable multi-level annotation of natural scenes in a large-scale dataset.

Self-calibration using pure rotation is a well-known technique and has been shown to be a reliable means for recovering intrinsic camera parameters. However, in practice, it is virtually impossible to ensure that the camera motion for this type of self-calibration is a pure rotation. In this paper, we present an error analysis of recovered intrinsic camera parameters due to the presence of translation. We derived closed-form error expressions for a single pair of images with nondegenerate motion; for multiple rotations for which there are no closed-form solutions, analysis was done through repeated experiments. Among others, we show that translation-independent solutions do exist under certain practical conditions. Our analysis can be used to help choose the least error-prone approach (if multiple approaches exist) for a given set of conditions.

In this paper, a novel method to learn the intrinsic object structure for robust visual tracking is proposed. The basic assumption is that the parameterized object state lies on a low dimensional manifold and can be learned from training data. Based on this assumption, firstly we derived the dimensionality reduction and density estimation algorithm for unsupervised learning of object intrinsic representation, the obtained non-rigid part of object state reduces even to 2 dimensions. Secondly the dynamical model is derived and trained based on this intrinsic representation. Thirdly the learned intrinsic object structure is integrated into a particle-filter style tracker. We will show that this intrinsic object representation has some interesting properties and based on which the newly derived dynamical model makes particle-filter style tracker more robust and reliable. Experiments show that the learned tracker performs much better than existing trackers on the tracking of complex non-rigid motions such as fish twisting with self-occlusion and large inter-frame lip motion. The proposed method also has the potential to solve other type of tracking problems.

In this paper, we propose a set of human perceptual cues used jointly to automatically detect image orientation. The cues used are: orientation of faces, position of the sky, brighter regions, and textured objects, and symmetry. We combine these cues in a Bayesian framework, and the photo acquiring model has been considered carefully as the prior knowledge of the image orientation. Results on more than a thousand different images provide a compelling argument that our approach is a viable one.

Practical content-based image retrieval systems require efficient relevance feedback techniques. Researchers have proposed many relevance feedback methods using quadratic-form distance metric as similarity measure and learning similarity matrix from feedback samples by linear transform. Existing linear approaches do not deal with data distribution in real image database very well. In this paper, a novel approach using binary component discriminant function (BCDF) is proposed by generalizing the original quadratic-form distance metric. The BCDF approach leams similarity measure nonlinearly by scatter criterion and distance criterion and deals with data distribution of feedback samples from real image database reasonably well. Experiments on a large database of 13,897 heterogeneous images demonstrated a remarkable improvement of retrieval precision compared with linear approaches.

The rapid increase of learning resources makes it difficult to search, manage and reuse. Using metadata is an efficacious way to solve this problem. With consistent descriptions of the characteristics of learning resources, searching becomes more specific and accurate, management becomes more simple and uniform, and sharing becomes more efficient and in-depth. The Learning Object Metadata Schema developed by IEEE P1484.12 is one of the most promising metadata approaches for describing learning resources, on which we developed a learning resource metadata management system (LRMMS). The system provides a platform for users to register, browse, search and evaluate learning resources. It is a decentralized framework with several metadata servers to provide services. The system supports distributed queries and the evaluation loop of learning resources. Users can search resources from different points of view, especially educational needs. We also made the interface as user-friendly as possible.

Proposes a robust pose estimation algorithm from 2D correspondences, which is a key issue of a 3D face modeling system from calibrated stereo sequences. The estimated rigid motion parameters are utilized to obtain the perspective projection of a generic face model, which is then matched with 2D clues extracted from the image under corresponding pose to decide the shape of a specified face. The main merits of our method are: (1) In order to obtain robust and accurate results under the situation of dramatic pose variation, we first evaluate the reliability of 2D tracker. Then after eliminating erroneous 2D correspondences, we refine the rigid motion parameters estimated between successive poses by performing a non-linear, batch estimator to compute the parameters of all poses in a clip of stereo sequences simultaneously. (2) Full automaticity is achieved by detecting and matching new features when there are not enough reliable 2D tracking results. Experiments show that this algorithm is accurate and robust, and help our system reach a satisfactory face modeling result.

In this paper a new condensation style contour tracking method called probabilistic dynamic contour (PDC) is proposed for lip tracking: a novel mixture dynamic model is designed to represent shape more compactly and to tolerate larger motions between frames, a measurement model is designed to include multiple visual cues. The proposed PDC tracker has the advantage that it is conceptually general but effectively suitable for lip tracking with the designed dynamic and measurement model. The new tracker improves the traditional condensation style tracker in three aspects: Firstly, the dynamic model is partially derived from the image sequence, so the tracker does not need to learn the dynamics in advance. Secondly, the measurement model is easy to be updated during tracking, which avoids modeling the foreground object in prior. Thirdly, to improve the tracker's speed, a compact representation of shape and a noise model are proposed to reduce the samples required to represent the posterior distribution. An experiment on lip contour tracking shows that the proposed method tracks contours robustly as well as accurately compared to the existing tracking method.

A software infrastructure is fundamental to a Smart Space. Previously proposed software infrastructures for Smart Space (SISS) did not sufficiently address the issue of performance and usability. A new solution, Smart Platform, which is focused on improving these aspects of a SISS, is presented in this paper. To optimize its intermodule communication performance, the stream-oriented communication is distinguished from the message-oriented ones, and a corresponding hybrid communication scheme is proposed. To improve the usability, a featured loose coupling structure, a straightforward Publish-and-Subscribe coordination model as well as a set of user-friendly deployment and development tools are developed. Besides, Smart Platform is intended as an open and generic SISS available for other research groups. To this end, XML-based message syntax and the open wire-protocol based architecture are adopted to make sharing research efforts more easily.

Different from multi-user database management systems, CSCW systems must support group awareness explicitly, namely that participants should perceive the presence of each other during the process of cooperation, which is essential to effective collaboration. The authors firstly analyze and compare several means of supporting sensibility, then investigate the model of collaborative design and suggest achieving group awareness by product data which forms the foundation of a relationship among designers. The information model and the shared workspace organization of CECAD (Collaborative Environment for Computer-Aided Design, a prototype system developed by the authors) are presented and its group awareness supporting mechanism is described in succession.

A face detection algorithm integrating template matching and support vector machines (SVM) is presented. Two types of templates: eyes-in-whole and face itself, are used for coarse filtering, and the SVM classifier is used for classification. A bootstrap method is used to collect non-face samples for SVM training under a template matching constrained subspace, which greatly reduces the complexity of training the SVM. Comparative experimental results demonstrate its effectiveness.

This paper addresses the subject of stereo matching between the corners of two perspectives. Similarity-based matching is prone to errors, and the existing algorithms reject outliers but never correct them. Frequently, in fact, the correct correspondence is not found at the single point with the largest similarity; but it lies among a few points with large value. So we propose a new algorithm, which first selects several candidates for each corner and then optimizes the whole match with global constraints. The algorithm increases remarkably not only the percentage of correctness, but also the number of correct matches. To expedite the optimization, we apply group evolution. A simple directional constraint is used as criteria for evaluation, which avoids the estimation of epipolar lines. The principles and applicable cases are presented. Results are provided for corners, retrieved both manually and automatically, in real images.

This paper, aiming at real-time gesture-controlled interaction, describes visual modeling, analysis, and recognition of continuous dynamic hand gestures. By hierarchically integrating multiple cues, a spatio-temporal appearance model and novel approaches are proposed for modeling and analysis of dynamic gestures respectively. At low level, fusion of flesh chrominance analysis and coarse image motion detection is employed to detect and segment hand gestures; at high level, parameters of the spatio-temporal appearance model are recovered by combining robust parameterized image motion estimation and hand shape analysis. The approach, therefore, fulfils real-time processing as well as high recognition rates. Without resorting to any special marks, twelve kinds of hand gestures can be recognized with average accuracy over 89%. A prototype system, gesture-controlled panoramic map browser is designed and implemented to demonstrate the usability of gesture-controlled interaction.

This paper presents a novel approach to stabilize the output of video camera installed on a moving vehicle in a rugged environment. A 2.5D interframe motion model is proposed so that the stabilization system can perform in situations where significant depth changes are present and the camera has both rotation and translation. Inertial motion filtering is proposed in order to eliminate the vibration of the video sequences with enhanced perceptual properties. The implementation of this new approach integrates four modules: pyramid-based motion detection, motion identification and 2.5D motion parameter estimation, inertial motion filtering, and affine-based motion compensation. The stabilization system can smooth unwanted vibrations or shakes of video sequences and achieve real-time speed. We test the system on IBM PC compatible machines and the experimental results show that our algorithm outperforms many algorithms which require parallel pipeline image processing machines.

This paper presents a novel approach for generating and analyzing epipolar plane images (EPIs) from video sequences taken from a moving platform subject to vibration so that the 3D model of an arbitrary scene can be constructed. Two problems are solved in our approach: (1) how to generate EPIs from video under a more general motion than a pure translation; (2) how to analyze the huge amount of data in the EPIs robustly and efficiently. For the first problem, a 3D image stabilization method is proposed which decouples the vibration from the vehicle's motion so that good EPIs and panoramic view images (PVIs) can be generated. For the second problem, we propose an efficient panoramic EPI analysis (PEPIA) method in which only one scanline of each EPI is processed. The PEPIA combines advantages of PVIs and EPIs and consists of three important steps: locus orientation detection, motion boundary localization, and occlusion/resolution recovery. The output of the PEPIA-a layered 3D panorama, is very useful in visual navigation and virtual reality modeling. Since camera calibration, image segmentation, feature extraction and matching are avoided, all the proposed algorithms are fully automatic and rather general. Results on real image sequences are given.

A new approach is presented to automatically build a dynamic and multi-resolution 360/spl deg/ panorama (DMP) from image sequences taken by a hand-held camera. A multi-resolution representation is built for the more interesting areas by means of camera zooming. The dynamic objects in the scene can be detected and represented separately. The DMP construction method is fast, robust and automatic, achieving 1 Hz in a 266 MHz PC.

In this paper, we present an approach to isolated gesture recognition, which uses as input the monocular sequence of dynamic gesturing with the single hand. We use the direct method of motion estimation, and realize gesture segmentation by image registration and warping based on robust A P-estimator and the dominant motion model. After extracting the spatial-temporal features of gestures, we perform dynamic time warping (DTW) to recognize 12 types of control gestures (6 for translation, 6 for rotation) replacing the 3-D mouse tool. A small demonstration system has been set to verify, our method, i.e. controlling a panorama image (set by mosaicing a sequence of standard "Garden" images) viewer with gesturing.

This paper presents the results of integrating omnidirectional view image analysis and a set of adaptive backpropagation networks to understand the outdoor road scene by a mobile robot. Both the road orientations used for robot heading and the road categories used for robot localization are determined by the integrated system, the road understanding neural networks (RUNN). Classification is performed before orientation estimation so that the system can deal with road images with different types effectively and efficiently. An omni-view image (OVI) sensor captures images with 360 degree view around the robot in real-time. The rotation-invariant image features are extracted by a series of image transformations, and serve as the inputs of a road classification network (RCN). Each road category has its own road orientation network (RON), and the classification result (the road category) activates the corresponding RON to estimate the road orientation of the input image. Several design issues, including the network model, the selection of input data, the number of the hidden units, and learning problems are studied. The internal representations of the networks are carefully analyzed. Experimental results with real scene images show that the method is fast and robust.

The authors present a novel approach using 2D spatio-temporal images for automatic traffic monitoring. A TV camera is mounted above the highway to monitor the traffic through two slice windows for each traffic lane. One slice window is along the lane and the other perpendicular to the lane axis. Two types of 2D spatio-temporal (ST) images are used in the system: the panoramic view image (PVI) and the epipolar plane image (EPI). The real-time vision system for automatic traffic monitoring, VISATRAM, an inexpensive system with a PC 486 and an image frame grabber has been tested with real road images. Not only can the system count the vehicles and estimate their speeds, but it can also classify the passing vehicles using 3D measurements (length, width and height). The VISATRAM works robustly under various light conditions including shadows in the day and vehicle lights at night, and automatically copes with the gradual and abrupt changes of the environment.

In this paper we model the problem of structure from motion as the range estimation with known motion. First, we approximate the motion within a reasonable time interval as a 3D translation and thus some image transformations are applied to convert an arbitrary motion to a 1D translation. Then we analyse the epipolar plane image in the Fourier domain to avoid the feature extraction and correspondence problems. Experimental results with real scene images have shown the efficiency and robustness of the approach.

Presents a new framework for detection of dynamic obstacles in the unstructured outdoor road environment by integrating binocular color image sequences. In the authors' system, color image segmentation, stereo obstacle detection, visual egomotion estimation, and moving object analysis, are all built-in task-oriented modules and hence are efficient and robust. Most of these functions can be performed in real-time. They are activated and integrated adaptively in the manner of neural parallel distributed processing (PDP). Experimental results are given to validate the authors' philosophy of so-called purposive vision, retinal mapping, adaptive integration and parallel distributed processing.>

A method is presented for the segmentation and description of 3-D objects in a range image. The image is segmented in two parallel ways: by region classification based on surface normal analysis and by edge detection using a relaxation process. The two results are combined by a small rule-based system to obtain a more reliable segmentation. Features of the 3D object at different levels are extracted and a surface attributes graph is constructed. This kind of view-independent description of 3D objects is effective and robust for object recognition and location. The objects considered are industrial parts composed of planes and quadric surfaces. The range image is provided by a laser scanner.>