Due to their complementary properties, inertial measurement units (IMU) and cameras are used in ego-motion tracking applications. For this, the relative rotation and displacement between the camera and IMU reference frames has to be known. There are established methods for the accurate estimation of the relative orientation, however, accurate estimation of the displacement is still a challenging problem. When this is not possible, one might resort to the alternative approach of fusing camera and gyroscope data only, as this does not require the displacement information. To be able to asses such alternatives, this paper presents a systematic methodology based on realistic simulations to analyze the effect of the camera - IMU displacement calibration error on tracking performance, and discusses in detailed simulations the dependency of tracker performance metrics on the camera - IMU displacement's magnitude and calibration error.

In a setup where camera measurements are used to estimate 3D egomotion in an extended Kalman filter (EKF) framework, it is well-known that inertial sensors (i.e., accelerometers and gyroscopes) are especially useful when the camera undergoes fast motion. Inertial sensor data can be fused at the EKF with the camera measurements in either the correction stage (as measurement inputs) or the prediction stage (as control inputs). In general, only one type of inertial sensor is employed in the EKF in the literature, or when both are employed they are both fused in the same stage. In this paper, we provide an extensive performance comparison of every possible combination of fusing accelerometer and gyroscope data as control or measurement inputs using the same data set collected at different motion speeds. In particular, we compare the performances of different approaches based on 3D pose errors, in addition to camera reprojection errors commonly found in the literature, which provides further insight into the strengths and weaknesses of different approaches. We show using both simulated and real data that it is always better to fuse both sensors in the measurement stage and that in particular, accelerometer helps more with the 3D position tracking accuracy, whereas gyroscope helps more with the 3D orientation tracking accuracy. We also propose a simulated data generation method, which is beneficial for the design and validation of tracking algorithms involving both camera and inertial measurement unit measurements in general.

This paper presents a hybrid method for face detection in color images. The well known Haar feature-based face detector developed by Viola and Jones (VJ), that has been designed for gray-scale images is combined with a skin-color filter, which provides complementary information in color images. The image is first passed through a Haar-Feature based face detector, which is adjusted such that it is operating at a point on its ROC curve that has a low number of missed faces but a high number of false detections. Then, using the proposed skin color post-filtering method many of these false detections can be eliminated easily. We also use a color compensation algorithm to reduce the effects of lighting. Our experimental results on the Bao color face database show that the proposed method is superior to the original VJ algorithm and also to other skin color based pre-filtering methods in the literature in terms of precision.

We present a Random Sampling Consensus (RANSAC) based training approach for the problem of speaker state recog-nition from spontaneous speech. Our system is trained and tested with the INTERSPEECH 2011 Speaker State Challenge corpora that includes the Intoxication and the Sleepiness Sub-challenges, where each sub-challenge defines a two-class clas-sification task. We aim to perform a RANSAC-based training data selection coupled with the Support Vector Machine (SVM) based classification to prune possible outliers, which exist in the training data. Our experimental evaluations indicate that utilization of RANSAC-based training data selection provides 66.32 % and 65.38 % unweighted average (UA) recall rate on the development and test sets for the Sleepiness Sub-challenge, respectively and a slight improvement on the Intoxication Sub-challenge performance.

We propose the use of the line spectral frequency (LSF) features for emotion recognition from speech, which have not been been previously employed for emotion recognition to the best of our knowledge. Spectral features such as mel-scaled cepstral coefficients have already been successfully used for the parameterization of speech signals for emotion recognition. The LSF features also offer a spectral representation for speech, moreover they carry intrinsic information on the formant structure as well, which are related to the emotional state of the speaker [4]. We use the Gaussian mixture model (GMM) classifier architecture, that captures the static color of the spectral features. Experimental studies performed over the Berlin Emotional Speech Database and the FAU Aibo Emotion Corpus demonstrate that decision fusion configurations with LSF features bring a consistent improvement over the MFCC based emotion classification rates.

We present a speech signal driven emotion recognition sys-tem. Our system is trained and tested with the INTERSPEECH 2009 Emotion Challenge corpus, which includes spontaneous and emotionally rich recordings. The challenge includes clas-sifier and feature sub-challenges with five-class and two-class classification problems. We investigate prosody related, spec-tral and HMM-based features for the evaluation of emotion recognition with Gaussian mixture model (GMM) based clas-sifiers. Spectral features consist of mel-scale cepstral coeffi-cients (MFCC), line spectral frequency (LSF) features and their derivatives, whereas prosody-related features consist of mean normalized values of pitch, first derivative of pitch and inten-sity. Unsupervised training of HMM structures are employed to define prosody related temporal features for the emotion recog-nition problem. We also investigate data fusion of different fea-tures and decision fusion of different classifiers, which are not well studied for emotion recognition framework. Experimen-tal results of automatic emotion recognition with the INTER-SPEECH 2009 Emotion Challenge corpus are presented. Index Terms: emotion recognition, prosody modeling 1.

This paper presents a framework for unsupervised video analysis in the context of dance performances, where gestures and 3D movements of a dancer are characterized by repetition of a set of unknown dance figures. The system is trained in an unsupervised manner using Hidden Markov Models (HMMs) to automatically segment multi-view video recordings of a dancer into recurring elementary temporal body motion patterns to identify the dance figures. That is, a parallel HMM structure is employed to automatically determine the number and the temporal boundaries of different dance figures in a given dance video. The success of the analysis framework has been evaluated by visualizing these dance figures on a dancing avatar animated by the computed 3D analysis parameters. Experimental results demonstrate that the proposed framework enables synthetic agents and/or robots to learn dance figures from video automatically.

We present a new framework for joint analysis of head gesture and speech prosody patterns of a speaker towards automatic realistic synthesis of head gestures from speech prosody. The proposed two-stage analysis aims to "learn" both elementary prosody and head gesture patterns for a particular speaker, as well as the correlations between these head gesture and prosody patterns from a training video sequence. The resulting audio-visual mapping model is then employed to synthesize natural head gestures from arbitrary input test speech given a head model for the speaker. Objective and subjective evaluations indicate that the proposed synthesis by analysis scheme provides natural looking head gestures for the speaker with any input test speech.

Various performing arts increasingly employ projection screens and other information displays as essential elements of the show. We present some possibilities and lessons learned for the interactiv, "immaterial" walk-through FogScreen in performing arts, analyze its suit for stage setups, and give some guidelines for show desig. We also describe our experiments on composing inter 3D graph for a FogScreen-based performance.

We describe a semi-automatic approach for segmenting a video sequence into spatio-temporal video objects in the presence of occlusion. Motion and shape of each video object is represented by a 2-D mesh. Assuming that the boundary of an object of interest is interactively marked on some keyframes, the proposed method finds the boundary of the object in all other frames automatically by tracking the 2-D mesh representation of the object in both forward and backward directions. A key contribution of the proposed method is automatic detection of covered and uncovered regions at each frame, and assignment of pixels in the uncovered regions to the object or background based on color and motion similarity. Experimental results are presented on two MPEG-4 test sequences and the resulting segmentations are evaluated both visually and quantitatively.

We propose a novel technique for tracking the visible boundary of a video object in the presence of occlusion. Starting with an initial contour that is interactively specified by the user and may be automatically refined by using intra-energy terms, the proposed technique employs piecewise contour prediction using local motion and color information on both sides of the contour segment, and contour snapping using scale-invariant intra-frame and inter-frame energy terms. The piecewise (segmented) nature of the contour prediction scheme and modeling of the motion on both sides of each contour segment enable accurate determination of whether and where the tracked boundary is occluded by another object. The proposed snake energy terms are associated with contour segments (as opposed to node points) and they are scale/resolution independent to allow multi-resolution contour tracking without the need to retune the weights of the energy terms at each resolution level. This facilitates contour prediction at coarse resolution and snapping at fine resolution with high accuracy. Experimental results are provided to illustrate the performance of the proposed occlusion detection algorithm and the novel snake energy terms that enable visible boundary tracking in the presence of occlusion.

We present a two-dimensional (2-D) mesh-based mosaic representation, consisting of an object mesh and a mosaic mesh for each frame and a final mosaic image, for video objects with mildly deformable motion in the presence of self and/or object-to-object (external) occlusion. Unlike classical mosaic representations where successive frames are registered using global motion models, we map the uncovered regions in the successive frames onto the mosaic reference frame using local affine models, i.e., those of the neighboring mesh patches. The proposed method to compute this mosaic representation is tightly coupled with an occlusion adaptive 2-D mesh tracking procedure, which consist of propagating the object mesh frame to frame, and updating of both object and mosaic meshes to optimize texture mapping from the mosaic to each instance of the object. The proposed representation has been applied to video object rendering and editing, including self transfiguration, synthetic transfiguration, and 2-D augmented reality in the presence of self and/or external occlusion. We also provide an algorithm to determine the minimum number of still views needed to reconstruct a replacement mosaic which is needed for synthetic transfiguration. Experimental results are provided to demonstrate both the 2-D mesh-based mosaic synthesis and two different video object editing applications on real video sequences.

This paper addresses 2-D mesh-based object tracking and mesh-based object mosaic construction for synthetic transfiguration of deformable video objects with deformable boundaries in the presence of another occluding object and/or self-occlusion. In particular, we update the 2-D triangular mesh model of a video object incrementally to account for the newly uncovered parts of the object as they are detected during the tracking process. Then, the minimum number of reference views (still images of a replacement object) needed to perform the synthetic transfiguration (object replacement and animation) is determined (depending on the complexity of the motion of the object-to-be-replaced), and the transfiguration of the replacement object is accomplished by 2-D mesh-based texture mapping in between these reference views. The proposed method is demonstrated by replacing an orange juice bottle by a cranberry juice bottle in a real video clip.

We propose a semi-automatic alpha-map generation method, where video object boundaries that are marked interactively on some key-frames are tracked in all other frames using only YUV data to generate alpha-maps. We use a 2-D triangular mesh to represent and track video objects. One of the major problems in alpha plane generation by object tracking is the detection and classification of uncovered regions due to object motion and self-occlusion (especially in the presence of out-of-plane rotations). The proposed method can identify uncovered regions and then classify them as belonging to the object of interest or not based on color and motion similarities.

Many interactive video and multimedia applications, such as interactive TV, augmented reality and bit stream editing, demand object-based video modeling. These applications require accurate tracking of the boundary, local motion and intensity variations of each object. This paper proposes a new approach to track deformable objects in video sequences, which combines deformable contours and 2-D mesh models while addressing partial occlusion, including self-occlusion. Experimental results demonstrate successful tracking of objects with deformable boundaries in the presence of self-occlusion.

We propose a method for tracking the motion and intensity variations of a 2-D mildly deformable image object using a hierarchical 2-D mesh model. The proposed method is applied to synthetic object transfiguration, namely, replacing an object in a real video clip with another synthetic or natural object via digital postprocessing. Successful transfiguration requires accurate tracking of both motion and intensity (contrast and brightness) variations of the object-to-be-replaced so that the replacement object can be rendered in exactly the same way from a single still picture. The proposed method is capable of tracking image regions corresponding to scene objects with nonplanar and/or mildly deforming surfaces, accounting for intensity variations, and is shown to be effective with real image sequences.

Synthetic object transfiguration is the problem of replacing an existing object in an image sequence with a new one via digital post-processing. Successful transfiguration requires accurate tracking of both motion and intensity variations of the existing object so that the new object can be rendered in exactly the same way. We propose a 2-D mesh tracking method that estimates, on a local basis, the motion and intensity variations of a region of interest. The proposed method is capable of accounting for illumination changes, tracking image regions corresponding to scene objects with non-planar and/or mildly deforming surfaces, and is shown to be effective in real-life situations.

Provides a theoretical analysis of the multigeneration properties of the MPEG video compression algorithms using the theory of generalized projections (GP). Multi-generations of an input video are obtained by successively compressing and decompressing it. The authors show that, under certain assumptions, successive generations can be interpreted as successive projections of the input video onto well-defined sets in a Hilbert space. Then, using the theory of GP, they establish that the multi-generations of I, P, and B pictures in the MPEG algorithms saturate.>

A computationally efficient multiframe LMMSE filtering algorithm, the motion-compensated multiframe (MCMF) Wiener filter, for restoring image sequences that are degraded by both blur and noise is proposed. MCMF Wiener filter applies to the cases where each frame of the ideal image sequence can be expressed as a globally shifted version of its previous frame. As opposed to single-frame filtering, the MCMF Wiener filter accounts for interframe (temporal) correlations as well as intraframe (spatial) correlations in restoring a given image sequence. The MCMF filter requires neither the explicit estimation of cross correlations among the frames, nor any matrix inversion. It accounts for the interframe correlations implicitly by using the estimated interframe motion information. The results of an extensive study on the performance and robustness of the proposed algorithm are presented.>

Computationally efficient multiframe Wiener filtering algorithms that account for both intraframe (spatial) and interframe (temporal) correlations are proposed for restoring image sequences that are degraded by both blur and noise. One is a general computationally efficient multiframe filter, the cross-correlated multiframe (CCMF) Wiener filter, which directly utilizes the power and cross power spectra of only NxN matrices, where N is the number of frames used in the restoration. In certain special cases the CCMF lends itself to a closed-form solution that does not involve any matrix inversion. A special case is the motion-compensated multiframe (MCMF) filter, where each frame is assumed to be a globally shifted version of the previous frame. In this case, the interframe correlations can be implicitly accounted for using the estimated motion information. Thus the MCMF filter requires neither explicit estimation of cross correlations among the frames nor matrix inversion. Performance and robustness results are given.

A 2-D autoregressive (AR) model is proposed which is not based on an underlying linear process representation, for matching-extrapolation of arbitrary bicumulant sequences of one-dimensional signals. This model employs the same number of coefficients as the number of given bicumulant samples, thus, makes it possible to extrapolate a given finite set of bicumulant samples to infinity while being consistent with, i.e., matching, all of the given samples. The computation of the model coefficients requires solving a system of linear equations. An algorithm is developed for the extrapolation of the given bicumulant samples based on the proposed 2-D AR model. An example is presented to compare the performance of the proposed method with other approaches for the extrapolation of a given set of bicumulant samples.>

The point spread function (PSF) of a defocused lens, in the continuous spatial coordinates, can be modeled using the principles of either geometrical optics or physical optics. The authors compare the PSFs and the corresponding optical transfer functions obtained on the basis of these two models under different amounts of blurs and different imaging sensor resolutions, in the discrete spatial domain. Different approximations for the discretization of the PSF are considered. The effect of using geometrical versus physical optics-based models on the quality of restored images is investigated experimentally in the case of images recorded by a charge-coupled device (CCD) camera.>

A novel signal representation, called a system with multiplicity K (SWM-K) is reviewed for exact modeling of a given polynomial bispectrum. An extension of this model is presented such that the extended model can simultaneously realize an arbitrary polynomial bispectrum and a rational power spectrum. In order to simultaneously realize arbitrary polynomial bispectra and rational power spectra, another linear shift-invariant system which is driven by a Gaussian input independent of the other K inputs is added to the SWM-K. Conditions for the existence of this composite model are stated. Simulation results are provided for an image processing example.>

Blur identification methods based on the bispectrum of an observed noisy and blurred image are proposed. Blur identification is addressed, in the case of uniform blurs, by zero-crossing detection in a particular cross section of the bispectrum of the observed image. The identification of general finite-impulse-response blurs is considered. The blurred image is represented by a nonminimum phase ARMA (autoregressive moving average) model whose parameters are identified using the complex bispectrum of the observed image. These methods are superior to previous methods which are based on the second-order statistics of the image, since the bispectrum is insensitive to additive, Gaussian noise in theory, and it contains the phase of the blur transfer function. Simulation results are provided.>

The mathematical framework to develop parametric image models based on higher-order statistics (HOS) is discussed. For non-Gaussian images, the model parameters will depend on the HOS if the model has a nonlinear form or if a fidelity criterion other than the mean square error (MSE) is utilized. Three new image models are proposed: (i) a nonlinear model based on the MSE criterion, (ii) a linear model under a criterion other than the MSE, and (iii) a nonlinear model using a criterion other than the MSE. These models are applied to predictive image coding, and the results are compared with those obtained by a linear model based on the MSE criterion.>

The authors address the problem of modeling a given higher order spectrum as that of the output of a linear time-invariant system driven by a higher order white random signal. This can be posed as a higher order spectrum factorization problem. The authors provide a theorem concerning the existence of such a factorization. A fast algorithm for efficient implementation of the factorization, if it exists, is then proposed. As applications, the authors present the problems of identification of non-minimum-phase linear time-invariant systems and phase reconstruction.>