This paper presents a multi-modal emotion recognition framework that is capable of estimating the human emotional state through analyzing and fusing a number of non-invasive external cues. The proposed framework consists of a set of data analysis, feature extraction and emotion recognition modules for processing heterogeneous sensory data (e.g., visual appearance and speech) and a novel probabilistic information fusion model to accurately estimate the human emotional state. Experimental results demonstrate that the proposed emotion recognition framework can automatically and robustly recognize human emotional states. Our results also proof that by fusing complementary information such as facial expression analysis and voice intonation analysis results, the emotion recognition performance can be boosted and outperform each individual modal analysis. The proposed emotion recognition framework can be integrated into existing Intelligent Tutoring Systems (ITSs) for improving the effectiveness of the learning systems by providing feedbacks to the ITSs.

This paper presents a novel adaptive pattern-driven approach for compressing large-area high-resolution terrain data. Utilizing a pattern-driven model, the proposed approach achieves efficient terrain data reduction by modeling and encoding disparate visual patterns using a compact set of extracted features. The feasibility and efficiency of the proposed technique were corroborated by experiments using various terrain datasets and comparisons with the state-of-the-art compression techniques. Since different visual patterns are separated and modeled explicitly during the compression process, the proposed technique also holds a great potential for providing a good synergy between compression and compressed-domain analysis.

This paper presents a multi-modal affect recognition system that is capable of effectively estimating human affective states through analyzing and fusing a number of non-invasive external cues. The proposed system consists of a probabilistic information fusion model based on the influence diagram and a set of data analysis, feature extraction and affect recognition modules for processing heterogeneous sensory data (e.g., visual appearance and speech) to accurately estimate human affective states. Experimental results demonstrate that the proposed affect recognition system can automatically and robustly recognize human affective states. Our results also prove that by fusing complementary information such as facial expression with voice intonation and/or body posture, the affect recognition performance can be boosted and outperform the performance based on each individual information. The proposed multi-modal affect recognition system can be integrated into existing remote health monitoring and rehabilitation systems to provide objective, non-intrusive, and persistent affect recognition, thus assisting patients' health monitoring and diagnosis in a natural living environment.

In this work a hierarchical fusion framework for melding multiple classifiers has been introduced, to obtain improved performance for classification problems. The fusion framework is hybrid in nature which allows for feature and decision level fusion while also being application and data agnostic. With a set of data features and a pool of trainable classifiers as input, the fusion framework utilizes a Genetic Algorithm (GA) with a modified chromosome structure to identify the appropriate choice of classifiers, select feature inputs for each classifier, and to determine the suitable hierarchical structure for a three layered hybrid classifier fusion scheme. The paper describes the workings of the framework and shows results of improved performance over individual classifiers and the majority voting scheme when applied to physiological condition classification.

We propose a new Bayesian network (BN) model for both automatic and interactive image segmentation. A multilayer BN is constructed from an oversegmentation to model the statistical dependencies among superpixel regions, edge segments, vertices, and their measurements. The BN also incorporates various local constraints to further restrain the relationships among these image entities. Given the BN model and various image measurements, belief propagation is performed to update the probability of each node. Image segmentation is generated by the most probable explanation inference of the true states of both region and edge nodes from the updated BN. Besides the automatic image segmentation, the proposed model can also be used for interactive image segmentation. While existing interactive segmentation (IS) approaches often passively depend on the user to provide exact intervention, we propose a new active input selection approach to provide suggestions for the user's intervention. Such intervention can be conveniently incorporated into the BN model to perform actively IS. We evaluate the proposed model on both the Weizmann dataset and VOC2006 cow images. The results demonstrate that the BN model can be used for automatic segmentation, and more importantly, for actively IS. The experiments also show that the IS with active input selection can improve both the overall segmentation accuracy and efficiency over the IS with passive intervention.

Chain graph (CG) is a hybrid probabilistic graphical model (PGM) capable of modeling heterogeneous relationships among random variables. So far, however, its application in image and video analysis is very limited due to lack of principled learning and inference methods for a CG of general topology. To overcome this limitation, we introduce methods to extend the conventional chain-like CG model to CG model with more general topology and the associated methods for learning and inference in such a general CG model. Specifically, we propose techniques to systematically construct a generally structured CG, to parameterize this model, to derive its joint probability distribution, to perform joint parameter learning, and to perform probabilistic inference in this model. To demonstrate the utility of such an extended CG, we apply it to two challenging image and video analysis problems: human activity recognition and image segmentation. The experimental results show improved performance of the extended CG model over the conventional directed or undirected PGMs. This study demonstrates the promise of the extended CG for effective modeling and inference of complex real-world problems.

We propose a unified graphical model that can represent both the causal and noncausal relationships among random variables and apply it to the image segmentation problem. Specifically, we first propose to employ Conditional Random Field (CRF) to model the spatial relationships among image superpixel regions and their measurements. We then introduce a multilayer Bayesian Network (BN) to model the causal dependencies that naturally exist among different image entities, including image regions, edges, and vertices. The CRF model and the BN model are then systematically and seamlessly combined through the theories of Factor Graph to form a unified probabilistic graphical model that captures the complex relationships among different image entities. Using the unified graphical model, image segmentation can be performed through a principled probabilistic inference. Experimental results on the Weizmann horse data set, on the VOC2006 cow data set, and on the MSRC2 multiclass data set demonstrate that our approach achieves favorable results compared to state-of-the-art approaches as well as those that use either the BN model or CRF model alone.

We propose a framework that can conveniently capture heterogeneous relationships among multiple random variables. The framework is formulated based on a hybrid probabilistic graphical model. It allows using both directed links and undirected links to capture various types of relationships. Based on this framework, we develop a multiscale hybrid model for image segmentation. The multiscale model systematically captures the spatial relationships and causal relationships among such image entities as regions, edges, and vertices at different scales. We further show how to parameterize such a hybrid model and how to factorize its joint probability distribution according to the global Markov properties. Based on this factorization, we exploit the Factor Graph theory to perform joint probabilistic inference and solve for the image segmentation problem.

We propose a Dynamic Bayesian Network (DBN) model for upper body tracking. We first construct a Bayesian Network (BN) to represent the human upper body structure and then incorporate into the BN various generic physical and anatomical constraints on the parts of the upper body. Unlike the existing upper body models, ours aims at handling physically feasible body motion rather than only some typical motion patterns. We also explicitly model part self-occlusion in the DBN model, which allows to automatically detect the occurrence of self-occlusion and to minimize the effect of measurement errors on the tracking accuracy due to occlusion. Moreover, our method can handle both 2D and 3D upper body tracking within the same framework. Using the DBN model, upper body tracking can be achieved through probabilistic inference over time.

The global shape prior knowledge has been exploited by many image segmentation approaches in order to improve segmentation results when there are such problems as occlusion, cluttering, low contrast edges, etc. We propose a global shape prior representation and incorporate it into a level set based segmentation framework. This global shape prior can effectively help remove the cluttered elongate structures and island-like artifacts in the segmentation. We experimentally compare the performance of our global shape prior with an extensively used global shape prior introduced in [3]. The experimental results show that our global shape prior averagely achieves 15% higher precision rates with comparable recall rates, which demonstrates the efficacy of the proposed shape prior.

For many computer vision problems, it is very important to produce the groundtruth data. Manual data labeling is labor-intensive and prone to the human errors, whereas fully automatic data labeling is not feasible and reliable. In this paper, we propose an interactive labeling technique for efficient and accurate data labeling. Constructed on a Bayesian Network (BN), the automatic image labeler produces an initial labeling of the image. A human then examines the initial labeling and makes minor corrections. The human corrections and the image measurements are then integrated by the BN framework to produce a refined labeling. We demonstrate the capability of this technique on labeling Facial Action Units.