The development of a two-stage approach for appraisal inference from automatically detected Action Unit (AU) intensities in recordings of human faces is described. AU intensity estimation is based on a hybrid approach fusing information from an individually fitted mesh model of the faces and texture information. Evaluation results for two datasets and a comparison against a state-of-the-art system, namely OpenFace are provided. In the second stage, the emotional appraisals novelty, valence and control are predicted from estimated AU intensities by linear regressions. Prediction performance is evaluated based on face recordings from a market research study, which were rated by human observers in terms of perceived appraisals. Predictions of valence and control from automatically estimated AU intensities closely match those obtained from manually coded AUs in terms of agreement with human observers, while novelty predictions lag somewhat behind. Overall, results highlight the flexibility and interpretability of a two-stage approach to emotion inference.

Pain sensation is essential for survival, since it draws attention to physical threat to the body. Pain assessment is usually done through self-reports. However, self-assessment of pain is not available in the case of noncommunicative patients, and therefore, observer reports should be relied upon. Observer reports of pain could be prone to errors due to subjective biases of observers. Moreover, continuous monitoring by humans is impractical. Therefore, automatic pain detection technology could be deployed to assist human caregivers and complement their service, thereby improving the quality of pain management, especially for noncommunicative patients. Facial expressions are a reliable indicator of pain, and are used in all observer-based pain assessment tools. Following the advancements in automatic facial expression analysis, computer vision researchers have tried to use this technology for developing approaches for automatically detecting pain from facial expressions. This paper surveys the literature published in this field over the past decade, categorizes it, and identifies future research directions. The survey covers the pain datasets used in the reviewed literature, the learning tasks targeted by the approaches, the features extracted from images and image sequences to represent pain-related information, and finally, the machine learning methods used.

Over the last few decades, there has been an increasing call in the field of computer vision to use machine-learning techniques for the detection, categorization, and indexing of facial behaviors, as well as for the recognition of emotion phenomena. Automated Facial Expression Analysis has become a highly attractive field of competition for academic laboratories, startups and large technology corporations. This paper introduces the new Actor Study Database to address the resulting need for reliable benchmark datasets. The focus of the database is to provide real multi-view data, that is not synthesized through perspective distortion. The database contains 68-minutes of high-quality videos of facial expressions performed by 21 actors. The videos are synchronously recorded from five different angles. The actors' tasks ranged from displaying specific Action Units and their combinations at different intensities to enactment of a variety of emotion scenarios. Over 1.5 million frames have been annotated and validated with the Facial Action Coding System by certified FACS coders. These attributes make the Actor Study Database particularly applicable in machine recognition studies as well as in psychological research into affective phenomena-whether prototypical basic emotions or subtle emotional responses. Two state-of-the-art systems were used to produce benchmark results for all five different views that this new database encompasses. The database is publicly available for non-commercial research.

In many domains of computer vision, such as medical imaging and facial image analysis, it is necessary to combine shape (geometric) and appearance (texture) information. In this paper, we describe a method for combining geometric and texture-based evidence for facial actions within a Kalman filter framework. The geometric evidence is provided by a face alignment method. The texture-based evidence is provided by a set of Support Vector Machines (SVM) for various Action Units (AU). The proposed method is a practical solution to the problem of fusing categorical probabilities within a Kalman filter based state estimation framework. A first performance evaluation on upper face AUs demonstrates the practical applicability of the proposed fusion method. The method is applicable to arbitrary imaging domains, apart from facial action estimation.

A novel system for facial expression recognition is presented. It is designed to assess valence (positive or negative feeling) in the context of market research applications. Other design parameters that were met are real-time capability and robust results even when using off-the-shelf equipment (standard web cam and lighting). The paper further presents an empirical study that attests the system satisfying predictive validity for market research applications as long as very small absolute valence values close to zero are not interpreted.

We present a backwards compatible high dynamic range video coding framework based on H.264/AVC. It allows to extract a standard low dynamic range (LDR) as well as high dynamic range (HDR) video from one compressed bit stream. A joint global and local inter-layer prediction method is proposed to reduce the redundancy between the LDR and HDR layers. It is based on a common color space which can represent HDR video data perceptually lossless. We show how the inter-layer prediction parameters can be estimated in a rate-distortion optimized way and efficiently encoded to reduce side information. Our evaluations demonstrate that the proposed framework performs best, compared to the state-of-the-art for arbitrary tone-mapping operators. W.r.t. simulcast it allows for up to 50% bit rate saving.

This paper presents a technique for the efficient compression of high dynamic range video (HDR) sequences. Such video sequences usually represent several orders of magnitude of real-world luminance intensity levels. Therefore, they are mostly stored in a floating-point represention. In order to obtain a coded representation that is bit stream compatible with the H.264/AVC video coding standard, the float-valued HDR values have to be mapped to a suitable integer representation first. The mapping proposed in this paper is adapted to the dynamic range of each video frame. Furthermore, to compensate for the associated dynamic contrast variation across frames, a weighted prediction method and quantization adaptation are introduced. The experiments show that the proposed method offers highly efficient HDR video compression. Only a fraction of the bit rate of a non-adaptive reference method is required to represent an HDR video sequence at the same quality.

A wavelet-based multiview video coding scheme is presented in this paper. It uses a 4-D wavelet transform, which is composed of a 1-D temporal wavelet transform, namely motion compensated temporal filtering, a 1-D view-directional wavelet transform, namely disparity compensated view filtering and a 2-D spatial wavelet transform. Since the presented framework can make use of the inherent scalability properties of the wavelet transforms involved, it allows full scalability of the coded bitstream in the temporal, view, spatial, and quality dimensions. Coding performance close to the H.264/advanced video coding based standard multiview video codec is shown. Enhancements of the view transform, in order to better account for brightness and color variations across views are introduced. Additionally, the use of a signal adaptive anisotropic wavelet packet (WP) transform as a generalization of WP transforms for the spatial decomposition is proposed. Both enhancements lead to a decrease of bit rate of up to 11% compared with the baseline version of the codec.

Scalable multiview video coding might become important in future immersive telecommunication scenarios. Therefore, we present a fully scalable multiview video coding framework that is based on wavelets. It offers scalable decoding of the bitstream in the time, view, spatial, and quality dimensions. Whereas its performance is already comparable to state-of-the-art multiview video coding at the full spatial resolution, the quality of the downscaled spatial resolution is not satisfying. After reviewing the reasons for this, we propose an extended coding scheme where a spatial wavelet transform is performed first and the high spatial resolution layers of the multiview video sequence are predicted from the lower ones in the wavelet domain. The structure and performance of this in-band prediction are analyzed and several enhancements are proposed. Finally, it can be shown that efficient spatial scalability can be achieved in this framework with only minor performance loss at the full resolution.

An approach to scalable multi-view video coding with joint best basis wavelet packets is examined in this paper. A 4-D wavelet transform is used to decorrelate the multi-view video data temporally, view-directionally, and spatially for efficient scalable compression. Motion compensated temporal filtering (MCTF) is used for temporal, and disparity compensated view filtering (DCVF) for view- directional decomposition. Adaptive wavelet packets as a generalized wavelet decomposition are presented for spatial decomposition. Two algorithms to find the best basis wavelet packets are evaluated and compared with classical dyadic wavelet transform: a low complexity entropy based best basis search and a search algorithm in a rate-distortion framework. In both cases, the joint best basis is determined for a group of frames rather than for each frame individually. Therefore, the rate to spend for the tree description is minimal while advantage is taken of the similarity of frames within a temporal-view-directional subband.

A wavelet-based approach to scalable multi-view video coding (MVC) is examined in this paper. A 4-D wavelet transform is used to decorrelate the multi-view video data temporally, view-directionally, and spatially for efficient compression. Motion compensated temporal filtering (MCTF) is applied to each video sequence of each camera to exploit temporal correlation and inter-view dependencies are exploited with disparity compensated view filtering (DCVF). Together with a 2-D spatial wavelet transform the 4-D wavelet transform is constituted. The open-loop structure of the decomposition, together with embedded quantization and coding, allows for the construction of a bitstream which is fully scalable in temporal, view-directional, spatial and quality dimension. Results for scalable decoding in all dimensions are given, as well as comparisons to simulcast coding and state-of-the-art non-scalable multi-view video coding. Further, a significant coding gain is shown when the sequences are pre-processed with an algorithm for cross-view illumination compensation based on histogram matching.

In this paper, we propose two wavelet-based frameworks which allow fully scalable multi-view video coding. Using a 4-D wavelet transform, both schemes generate a bitstream that can be truncated to achieve a temporally, view-directionally, and/or spatially downscaled representation of the coded multi-view video sequence. Well-known wavelet-based scalable coding schemes for single-view video sequences have been adopted and extended to match the specific needs of scalable multi-view video coding. Motion compensated temporal filtering (MCTF) is applied to each Video sequence of each camera to exploit temporal correlation and inter-view dependencies are exploited with disparity compensated view filtering (DCVF). A spatial wavelet transform is utilized either before and after temporal-view-directional decomposition (2D+T+V+2D scheme) or only after the temporal-view-directional decomposition (T+V+2D scheme) for spatial decorrelation. The influence of the two different approaches on spatial scalability is shown in this paper as well as the superior coding efficiency of both codecs compared with simulcast coding.

In this paper, a novel framework for scalable multi-view video coding is described. A well known wavelet based scalable coding scheme for single-view video sequences has been adopted and extended to match the specific needs of scalable multi-view video coding. Motion compensated temporal filtering (MCTF) is applied to each video sequence of each camera. The use of a wavelet lifting structure guarantees perfect invertibility of this step, and as a consequence of its open-loop architecture, SNR and temporal scalability are attained. Correlations between the temporal subbands of adjacent cameras are reduced by a novel disparity compensated view filtering (DCVF), method which is also lifting based and open-loop to enable view scalability. Spatial scalability and entropy coding are achieved by the JPEG2000 spatial wavelet transform and EBCOT coding, respectively. Rate allocation along the temporal-view-filtered subbands is done by means of an RD-optimal algorithm. Experimental results show the high scaling capability in terms of SNR, temporal and view scalability