We have recently developed a real time automatic access control system suing both face and voice identification. We show that combining these two modalities achieves high recognition accuracy, especially in difficult recognition scenarios. Our system also supports online training of the user models, i.e., a person who has not been registered in the database can be added on the spot in real time. This system is composed of a face and eye corners detection module, a face recognition module, a speaker identification module, and a user interface module. We present the algorithm of each module with more emphasis on our information maximization based face and eye corners detection and user interface design. We discuss some issues related to the integration of components into the total system. We report the experimental results of the system performance together with comparison with those of some other systems.

Robust face recognition is an important research area that has a great impact in the field of surveillance and human-computer interactions in general. This paper addresses how the concept of motion pursuit can bridge the alignment information with imagery similarities to form a joint decision for the purpose of robust face recognition. More explicitly, we perform multi-scale correspondence matching in a wavelet domain and utilize a simple motion measure in conjunction with wavelet response matching to form a discriminative face similarity measure. We demonstrate the efficacy of the proposed approach with a large consumer image database of diverse image capture conditions.

In this paper an improved face detection method based on our previous information-based maximum discrimination approach is presented that maximizes the discrimination between face and non-face examples in a training set without using color or motion information. A short review of our previous method is given together with a description of a recent improvement of its detection speed. A person identification system has been developed that performs multi-modal person identification in real-time video based on this newly improved face detection method together with speaker identification.

This paper introduces a novel adaptive texture feature selection algorithm for tracking. Specifically, we provide a statistical wavelet basis paradigm to maximally separate statistical characteristics of the object-in-interest and its background. The algorithm is based upon nonlinearly selecting basis elements out of dual dictionaries in an iterative fashion to continually improve a cost function that is suitable for tracking. We demonstrate that such a selection is effective with several difficult sequences that are affected by lighting changes, occlusion and background motion.

We investigate a method for classification of patterns with temporal support. This method combines the ability of a nonlinear-kernel based classifier (in the form of a support vector machine) to discriminate and the ability of a first order Markov chain to model temporal transitions. We apply this to the task of classifying motion picture soundtrack. Experiments with classification of the soundtrack into speech and non-speech audio patterns reveal improvement in classification performance using this proposed method over HMM-based classification as well as SVM-based classification. Using a normalized margin obtained from the SVM and mapping it to a non-negative confidence measure bounded by 1, we attempt to alter the classification of patterns close to the separating boundary, by using the constraints on the transition between the two classes. Sound track classification with semantic classes can help browse and index a video efficiently.

A necessary capability for content-based retrieval is to support the paradigm of query by example. Most systems for video retrieval support queries using image sequences only. We present an algorithm for matching multimodal (audio-visual) patterns for the purpose of content-based video retrieval. The novel ability of our approach to use the information content in multiple media coupled with a strong emphasis on temporal similarity differentiates it from the state-of-the-art in content-based retrieval. At the core of the pattern matching scheme is a dynamic programming algorithm, which leads to a significant improvement in performance. Coupling the use of audio with video this algorithm can be applied to grouping of shots based on audio-visual similarity. We also support relevance feedback. The user can provide feedback to the system, by choosing clips, which are closer to the user's desired target. The system then automatically adjusts the relative weights or relevance of the media and fetches different sets of target clips accordingly. It is our observation that a few iterations of such feedback are generally sufficient, for retrieving the desired video clips.

Video can be viewed as the integration of several heterogeneous media interwoven in a temporally close-coupled fashion. To support the retrieval of video segments under the query-by-example scenario, we purport in this paper a post-integration model that integrates low-level media types to identify visually/auditorily similar video segments. The model also allows flexible relevance feedback on the user end to further improve the speed and accuracy of searching in a video database. As its name implies, the post-integration model first treats models of the underlying media as independent processes and then combines distance scores from each of the underlying media at the later stage. This decoupling allows the usage of efficient algorithms to fast and accurately compare thousands of video clips in a moderate-to-large-sized video database. It also naturally facilitates improved user interaction by fast dynamic weight adjustment on the media distance scores in a multiple query succession. We explicit our models and dynamic weight adjustment schemes in this paper and offer demonstration with a working system on a large video database set.

We present a novel approach for learning patterns (sub-images) shared by multiple images without prior knowledge about the number and the positions of the patterns in the images. The patterns may undergo kinds of rigid and non-rigid transformations. To reduce the searching space, the images are pre-segmented and represented by attribute relation graphs (ARGs). The problem is then formulated as learning the isomorphic subgraph, called pattern ARG (PARG) from multiple sample ARGs (SARG) with regard to the attribute similarity and the relation similarity. An inexact graph-matching algorithm is proposed to establish the correspondence between each SARG and the PARG. Inexact graph matching and model editing based on Bayes' decision rule are incorporated into Generalized Expectation and Maximization (GEM) algorithm. The modified GEM algorithm combines soft decisions and hard decisions together to learn both the appearance and the structure of the PARG. In the experiments, the learned PARG successfully captures the appearance and spatial information of the concept shared by the images.

As the proliferation of compressed video sequences in MPEG formats continues, the ability to perform video analysis directly in the compressed domain becomes increasingly attractive. The availability of motion vectors and pixel values in coded forms can indirectly provide motion and intensity information for object analysis, avoiding the need to re-perform motion estimation. Albeit that the embedded motion field is contaminated with matching modeling errors and measurement errors, we will illustrate several motion field filtering and correction techniques to combat with noisy motion fields. We strive to reconstruct smooth true motion fields with a minimal amount of decoding, reducing computational resource and time requirement. In this paper, we describe the whole moving object extraction system with the general framework and component designs and show their effectiveness with two test sequences.

Camera motion estimation is crucial in video analysis and in object tracking query systems when the motion need to be neutralized before object analysis. In today's ever growing amount of video data provided in compressed formats namely MPEG-1 and MPEG-2, it increasingly makes more sense to perform camera estimation in the compressed domain. Much work has gone into the uncompressed domain, but the time to decompress and analyze is simply too great for population of large video databases. This paper presents a recursive outlier-rejecting least square algorithm for parametric camera estimation in MPEG-1 and MPEG-2 domain. The algorithm has a very low time complexity, results show that it works much faster than real-time playback rate and consumes little system resource. Experiments on synthesized video clips and real world video clips show that the algorithm is effective. Experiments are also none on a large set of real-world video clips to analyze the performance and a query system is built in the process.