This paper presents a demonstration showcasing the capabilities of the upcoming MPEG-I standards on Haptics and Scene Description. The demonstration is a sensorial immersive experience using a virtual reality headset, haptic-enabled controllers and a haptic vest. The proposed experience is an interactive go-kart race-like game where interactions with the environment will trigger spatialized haptic feedback to provide an enhanced sense of immersion. By implementing the MPEG-I standards on Haptics and Scene Description, and by designing this demonstration using exclusively these standards, this work showcases part of the research and standardization efforts produced through the MPEG ecosystem.

While digital humans are key aspects of the rapidly evolving areas of virtual reality, gaming, and online communications, many applications would benefit from using digital personalized (stylized) representations of users, as they were shown to highly increase immersion, presence and emotional response. In particular, depending on the target application, one may want to look like a dwarf or an elf in a heroic fantasy world, or like an alien on another planet, in accordance with the style of the narrative. While creating such virtual replicas requires stylization of the user’s features onto the virtual character, no formal study has however been conducted to assess the ability to recognize stylized characters. In this paper, we present a perceptual study investigating the effect of the degree of stylization on the ability to recognize an actor, and the subjective acceptability of stylizations. Results show that recognition rates decrease when the degree of stylization increases, while acceptability of the stylization increases. These results provide recommendations to achieve good compromises between stylization and recognition, and pave the way to new stylization methods providing a tradeoff between stylization and recognition of the actor.

Electroencephalography (EEG)-based emotion recognition is currently a hot issue in the affective computing community. Numerous studies have been published on this topic, following generally the same schema: 1) presentation of emotional stimuli to a number of subjects during the recording of their EEG, 2) application of machine learning techniques to classify the subjects' emotions. The proposed approaches vary mainly in the type of features extracted from the EEG and in the employed classifiers, but it is difficult to compare the reported results due to the use of different datasets. In this paper, we present a new database for the analysis of valence (positive or negative emotions), which is made publicly available. The database comprises physiological recordings and 257-channel EEG data, contrary to all previously published datasets, which include at most 62 EEG channels. Furthermore, we reconstruct the brain activity on the cortical surface by applying source localization techniques. We then compare the performances of valence classification that can be achieved with various features extracted from all source regions (source space features) and from all EEG channels (sensor space features), showing that the source reconstruction improves the classification results. Finally, we discuss the influence of several parameters on the classification scores.

Deep neural networks have been recently proposed to solve video interpolation tasks. Given a past and future frame, such networks can be trained to successfully predict the intermediate frame(s). In the context of video compression, these architectures could be useful as an additional inter-prediction mode. Current inter-prediction methods rely on block-matching techniques to estimate the motion between consecutive frames. This approach has severe limitations for handling complex non-translational motions, and is still limited to block-based motion vectors. This paper presents a deep frame interpolation network for video compression aiming at solving the previous limitations, i.e. able to cope with all types of geometrical deformations by providing a dense motion compensation. Experiments with the classical bi-directional hierarchical video coding structure demonstrate the efficiency of the proposed approach over the traditional tools of the HEVC codec.

We present a new method for reconstructing a 4D light field from a random set of measurements. A 4D light field block can be represented by a sparse model in the Fourier domain. As such, the proposed algorithm reconstructs the light field, block by block, by selecting frequencies of the model that best fits the available samples, while enforcing orthogonality with the approximation residue. The method achieves a very high reconstruction quality, in terms of Peak Signal-to-Noise Ratio (PSNR). Experiments on several datasets show significant quality improvements of more than 1dB compared to state-of-the-art algorithms.

Video codecs are primarily designed assuming that rigid, block-based, two-dimensional displacements are suitable models to describe the motion taking place in a scene. However, translational models are not sufficient to handle real world motion types such as camera zoom, shake, pan, shearing or changes in aspect ratio. We present here a region-based interprediction scheme to compensate such motion. The proposed mode is able to estimate multiple homography models in order to predict complex scene motion. We also introduce an affine photometric correction to each geometric model. Experiments on targeted sequences with complex motion demonstrate the efficiency of the proposed approach compared to the state-of-the-art HEVC video codec.

Current virtual reality systems enable users to explore virtual worlds, fully embodied in avatars. This new type of immersive experience requires specific authoring tools. The traditional ones used in the movie and the video games industries were modified to support immersive visual and audio content. However, few solutions exist to edit haptic content, especially when the whole user's body is involved. To tackle this issue we propose HFX Studio, a haptic editor based on haptic perceptual models. Three models of pressure, vibration and temperature were defined to allow the spatialization of haptic effects on the user's body. These effects can be designed directly on the body (egocentric approach), or specified as objects of the scene (allocentric approach). The perceptual models are also used to describe capabilities of haptic devices. This way the created content is generic, and haptic feedback is rendered on the available devices. The concept has been implemented with the Unity®game engine, a tool already used in VR production. A qualitative pilot user study was conducted to analyze the usability of our tool with expert users. Results shows that the edition of haptic feedback is intuitive for these users.

Thanks to the increasing number of images stored in the cloud, external image similarities can be leveraged to efficiently compress images by exploiting inter-images correlations. In this paper, we propose a novel image prediction scheme for cloud storage. Unlike current state-of-the-art methods, we use a semi-local approach to exploit inter-image correlation. The reference image is first segmented into multiple planar regions determined from matched local features and super-pixels. The geometric and photometric disparities between the matched regions of the reference image and the current image are then compensated. Finally, multiple references are generated from the estimated compensation models and organized in a pseudo-sequence to differentially encode the input image using classical video coding tools. Experimental results demonstrate that the proposed approach yields significant rate-distortion performance improvements compared with the current image inter-coding solutions such as high efficiency video coding.

Quadrotor drones equipped with high-quality cameras have rapidly raised as novel, cheap, and stable devices for filmmakers. While professional drone pilots can create aesthetically pleasing videos in short time, the smooth—and cinematographic—control of a camera drone remains challenging for most users, despite recent tools that either automate part of the process or enable the manual design of waypoints to create drone trajectories. This article moves a step further by offering high-level control of cinematographic drones for the specific task of framing dynamic targets. We propose techniques to automatically and interactively plan quadrotor drone motions in dynamic three-dimensional (3D) environments while satisfying both cinematographic and physical quadrotor constraints. We first propose the Drone Toric Space , a dedicated camera parameter space with embedded constraints, and derive some intuitive on-screen viewpoint manipulators. Second, we propose a dedicated path planning technique that ensures both that cinematographic properties can be enforced along the path and that the path is physically feasible by a quadrotor drone. At last, we build on the Drone Toric Space and the specific path planning technique to coordinate the motion of multiple drones around dynamic targets. A number of results demonstrate the interactive and automated capacities of our approaches on different use-cases.

With the explosion of Virtual Reality technologies, the production and usage of omni directional images (a.k.a 360 images) is presenting new challenges in the domains of compression, transmission and rendering. The evaluation of the quality of images generated by these technologies is therefore paramount. As the exploration of 360 images within a Head-Mounted Display (HMD) is non-uniform, current state of the art proposes a saliency weighting of distortions (between a reference and an impaired version) thus allowing us to highlight impairments in frequently attended regions. So far, saliency maps have been generated by tracking head motion alone, and consider that the view-port orientation alone is sufficient to determine saliency. The added value of eye gaze tracking within the viewport has not been studied in this domain yet. In this work, an eye-tracking experiment is performed using a HMD and is followed by subsequent gaze analysis to appreciate the visual attention behavior within a view-port. Results suggest that most eye-gaze fixations are rather far away from the center of the viewport. Across contents and observers, gaze fixations are quasi-isotropically distributed in orientation. The average location of gaze fixation (across contents and observers) from the center of the view-port varies between 14 and 20 visual degrees and these values correspond to a shift in retina beyond the para and peri fovea, into the extra-perifoveal region. A saliency weighting model based on foveation, centered at the middle of the view-port seems to be a correct assumption in only 2.5% of the overall scenarios and is consequently questionable. Therefore there is a need to refine saliency modeling and weighting for quality assessment in case of panoramic viewing.

The papers in this special section focus on commercial applications for affective computing. One of the main goals of affective computing is to create machines which can adapt to users’ emotions in order to produce more natural and efficient interaction. Emotion recognition is thus a central component of the field, and is based on a variety of measurements (facial expressions, speech, gait patterns, physiology, eye tracking, etc.) that are analyzed using advanced pattern recognition techniques. Furthermore, researchers and entrepreneurs have identified countless possible applications of affect-aware technology, from health and driver monitoring to exercise [6] and computer game adaptation. However, although great scientific advances have been made and many applications have been proposed, only few robust implementations have been presented or validated, and commercial adoption of affect-aware technology has been marginal. One rare example of a successful application is the smile detector used in digital cameras to automatically take pictures when the subject is smiling. This weak adoption of the technology can be attributed to several unsolved challenges in the domain of affective computing.

In this paper, we propose a novel scheme for scalable image coding based on the concept of epitome. An epitome can be seen as a factorized representation of an image. Focusing on spatial scalability, the enhancement layer of the proposed scheme contains only the epitome of the input image. The pixels of the enhancement layer not contained in the epitome are then restored using two approaches inspired from local learning-based super-resolution methods. In the first method, a locally linear embedding model is learned on base layer patches and then applied to the corresponding epitome patches to reconstruct the enhancement layer. The second approach learns linear mappings between pairs of co-located base layer and epitome patches. Experiments have shown that the significant improvement of the rate-distortion performances can be achieved compared with the Scalable extension of HEVC (SHVC).

Between the recent popularity of virtual reality (VR) and the development of 3D, immersion has become an integral part of entertainment concepts. Head-mounted Display (HMD) devices are often used to afford users a feeling of immersion in the environment. Another technique is to project additional material surrounding the viewer, as is achieved using cave systems. As a continuation of this technique, it could be interesting to extend surrounding projection to current television or cinema screens. The idea would be to entirely fill the viewer's field of vision, thus providing them with a more complete feeling of being in the scene and part of the story. The appropriate content can be captured using large field of view (FoV) technology, using a rig of cameras for 110 to 360 capture, or created using computergenerated images. The FoV is, however, rather limited in its use for existing (legacy) content, achieving between 36 to 90 degrees () field, depending on the distance from the screen. This paper seeks to improve this FoV limitation by proposing computer vision techniques to extend such legacy content to the peripheral (extrafoveal) vision without changing the original creative intent or damaging the viewer's experience. A new methodology is also proposed for performing user tests in order to evaluate the quality of the experience and confirm that the sense of immersion has been increased. This paper thus presents: i) an algorithm to spatially extend the video based on human vision characteristics, ii) its subjective results compared to state-of-the-art techniques, iii) the protocol required to evaluate the quality of the experience (QoE), and iv) the results of the user tests.

Thanks to the increasing number of images stored in the cloud, external image redundancies can be leveraged to efficiently compress images by exploiting inter-images correlations. In this paper, we propose a novel cloud-based image coding scheme. Unlike current state-of-the-art systems, our method relies on a data dimensionality reduction technique. A global compensation is associated to a locally-weighted template matching compensation method to predict a reference frame, to be then differential-coded with classic video coding tools. Experimental results demonstrate that the proposed approach yields significant rate-distortion performance improvements compared to current image coding solutions.

This paper describes a novel scheme to reduce the quantization noise of compressed videos and improve the overall coding performances. The proposed scheme first consists in clustering noisy patches of the compressed sequence. Then, at the encoder side, linear mappings are learned for each cluster between the noisy patches and the corresponding source patches. The linear mappings are then transmitted to the decoder where they can be applied to perform de-noising. The method has been tested with the HEVC standard, leading to a bitrate saving of up to 9.63%.

We describe a novel epitomic image representation scheme that factors a given image content into a condensed epitome and a low-resolution image to reduce the memory space for images. Given an input image, we construct a condensed epitome such that all image patches can successfully be reconstructed from the factored representation by means of an optimized neighbor-embedding strategy. Under this new scope of epitomic image representations aligned with the manifold sampling assumption, we end up a more generic epitome learning scheme with increased optimality, compactness, and reconstruction stability. We present the performance of the proposed method for image and video up-scaling (super-resolution) while extensions to other image and video processing are straightforward.

This paper describes a novel image coding scheme based on epitome inpainting. An epitome containing a factorized texture representation of the image is first coded and transmitted. The decoded epitome is then inpainted by propagating its structure and texture with extended H.264 Intra directional prediction modes and advanced neighbor embedding methods. The blocks to be inpainted are processed according to an adaptive scanning order which adapts to the image content and gives a higher priority to the blocks having strong structures. The inpainting step has been incorporated in the coding loop of an H.264 encoder, leading to a bitrate saving of up to 48.12% with respect to H.264 intra.

We propose an exemplar-based super-resolution algorithm based on sparsity constrained neighbor-embeddings of local image patches. We extract exemplar patch pairs from as little as the given low-resolution image, and we rely on local geometric similarities of low-and high-resolution patch spaces. While sparsely coding the local geometry with a greedy patch selection method, we refine our solution by iteratively updating the obtained high-resolution image. We finally apply an adaptive back-projection to ensure the global consistency. Our experimental results indicate promising performance on synthesizing natural looking textures and sharp edges when compared to other super-resolution methods from the literature.

We propose here an objective study of the emotional impact of a movie on an audience. An affective benchmarking solution is introduced making use of a low-intrusive measurement of the well-known Electro Dermal Answer. A dedicated processing of this biosignal produces a time-variant and normalized affective signal related to the significant excitation variations of the audience. Besides the new methodology, the originality of this paper stems from the fact that this framework has been tested on a real audience during regular cinema shows and a film festival, for five different movies and a total of 128 audience members.

The ultimate goal of the movie industry is to create an emotion from a story. However it is difficult to measure this emotional impact on an user. Questionnaires or other audience measurement techniques are often used, but with well known biases difficult to remove. We propose here an objective solution based on the viewer's physiological response (aka the EDA or GSR signals). The demonstration consists of an on the fly and live capture of an user's physiological signals, corresponding to his unconscious body reaction, and the real-time computing and display of the corresponding user's "affective profile".

Image prediction methods based on data dimensionality reduction techniques have been recently introduced for still images. These techniques have been proven efficient, especially when used in a H.264 framework. This paper introduces a natural extension of this work from spatial prediction to temporal prediction. Locally Linear Embedding and Optimized Map-Aided Locally Linear Embedding methods are adapted within H.264 in order to improve inter image prediction. The resulting prediction methods are compared with the H.264 motion estimation/compensation method, template matching methods and Adaptive Interpolation Filters, and are shown to bring significant Rate-Distortion performance improvements (up to 20.21 % rate saving with respect to H.264).

We describe a self-content single-image super-resolution algorithm based on multi-scale neighbor embeddings of small image patches. Given an input low-resolution patch, we gradually expand its size by relying on local geometric similarities of low- and high-resolution patch spaces under small scaling factors. We characterize the local geometry with K-similar patches taken from an exemplar set and we collect exemplar patch pairs from the input image and its appropriately rescaled versions. While ensuring local images compatibility with an optimization on K, we satisfy image smoothness by patch overlapping. We further enforce global consistency through an adaptive back-projection. Our experimental results show better performance on synthesizing natural looking textures and sharp edges with less artifacts when compared to other methods.

This paper presents a novel approach for gaze estimation using only a low-cost camera and requiring no calibration. The main idea is based on the center-bias property of human gaze distribution to get a coarse estimate of the current gaze position as well as benefit from temporal information to enhance this rough gaze estimate. Firstly, we propose a method for detecting the eye center location and a mapping model based on the center-bias effect to convert it to gaze position. This initial gaze estimate then serves to construct the likelihood model of the eye-appearance. The final gaze position is estimated by fusing the likelihood model with the prior information obtained from previous observations on the basis of the particle filtering framework. Extensive experiments demonstrate the good performance of the proposed system with an average estimation error of 3.43° which outperforms state-of-the-art methods. Furthermore, the low complexity of the proposed system makes it suitable for real-time applications.

This paper describes new image prediction methods based on neighbor embedding (NE) techniques. Neighbor embedding methods are used here to approximate an input block (the block to be predicted) in the image as a linear combination of K nearest neighbors. However, in order for the decoder to proceed similarly, the K nearest neighbors are found by computing distances between the known pixels in a causal neighborhood (called template) of the input block and the co-located pixels in candidate patches taken from a causal window. Similarly, the weights used for the linear approximation are computed in order to best approximate the template pixels. Although efficient, these methods suffer from limitations when the template and the block to be predicted are not correlated, e.g., in non homogenous texture areas. To cope with these limitations, this paper introduces new image prediction methods based on NE techniques in which the K-NN search is done in two steps and aided, at the decoder, by a block correspondence map, hence the name map-aided neighbor embedding (MANE) method. Another optimized variant of this approach, called oMANE method, is also studied. In these methods, several alternatives have also been proposed for the K-NN search. The resulting prediction methods are shown to bring significant rate-distortion performance improvements when compared to H.264 Intra prediction modes (up to 44.75% rate saving at low bit rates).

Template matching has been shown to outperform the H.264 prediction modes for Intra video coding thanks to better spatial prediction and no additional ancillary data to transmit. The method indeed works well when the template and the block to be predicted are highly correlated, e.g., in homogenous image areas, however, it obviously fails in areas with non homogeneous textures. This paper explores the idea of using a block-matching intra prediction algorithm which, thanks to a Rate-Distorsion (RD) based decision mechanism, will naturally be used in image areas when template matching (TM) fails. This new method offers a significant coding gain compared to H.264 Intra prediction modes and the template matching based prediction. Indeed, the TM-based algorithm and the proposed hybrid algorithm lead, with the Bjontergaard measure, to rate gains of up to respectively 38.02% and 48.38% at low bitrates when compared with H.264 Intra only.

Image prediction methods based on data dimensionality reduction techniques have been introduced in [1]. Although efficient, these methods suffer from limitations when the block to be predicted and its neighborhood (or template) are not correlated, e.g. in non homogenous texture areas. To cope with these limitations, this paper introduces new image prediction methods based on locally linear embedding (LLE) technique in which the required K-NN search is aided, at the decoder, by a block correspondence map, hence the name Map-Aided Locally Linear Embedding (MALLE) method. Another optimized variant of this approach, called oMALLE method, is also studied. The resulting prediction methods are shown to bring significant Rate-Distortion (RD) performance improvements when compared to H.264 Intra prediction modes (up to 40.78 % rate saving at low bit rates).

Identifying violent scenes in a movie may be of high interest as soon as the associated content has to be shown to a specific audience, as children for instance. However, defining what a violent scene is as well as extracting the violent excerpts from the only audiovisual cue are two hard tasks. In this article, we propose a pilot study to evaluate the interest of an approach based on the use of the electrodermal activity (EDA) to address these problems in an objective and human-centered manner. Assuming a consumer context, we especially focus on the use of a commercial sensor to capture the EDA. A comparison with a more professional device is initially performed to validate the accuracy of the commercial sensor. Two main aspects of the violent scene impact measurement with the EDA are then discussed. To tackle the violent scene detection problem, a methodology to correlate the EDA information with manual annotations of violent excerpts is first proposed. The user sensitivity to violence is then addressed and an approach to identify different profiles in the audience is qualitatively proposed. The advantages and limitations of such an approach are discussed and potential improvements are finally proposed.

We introduce a novel way of simulating sensations of motion which does not require an expensive and cumbersome motion platform. Multiple force-feedbacks are applied to the seated user's body to generate a sensation of motion experiencing passive navigation. A set of force-feedback devices such as mobile armrests or headrests are arranged around a seat so that they can apply forces to the user. We have dubbed this new approach HapSeat. A proof of concept has been designed which uses three low-cost force-feedback devices, and two control models have been implemented. Results from the first user study suggest that subjective sensations of motion are reliably generated using either model. Our results pave the way to a novel device to generate consumer motion effects based on our prototype.

In this paper, we propose a methodology to build a real-time affect detector dedicated to video viewing and entertainment applications. This detector combines the acquisition of traditional physiological signals, namely, galvanic skin response, heart rate, and electromyogram, and the use of supervised classification techniques by means of Gaussian processes. It aims at detecting the emotional impact of a video clip in a new way by first identifying emotional events in the affective stream (fast increase of the subject excitation) and then by giving the associated binary valence (positive or negative) of each detected event. The study was conducted to be as close as possible to realistic conditions by especially minimizing the use of active calibrations and considering on-the-fly detection. Furthermore, the influence of each physiological modality is evaluated through three different key-scenarios (mono-user, multi-user and extended multi-user) that may be relevant for consumer applications. A complete description of the experimental protocol and processing steps is given. The performances of the detector are evaluated on manually labeled sequences, and its robustness is discussed considering the different single and multi-user contexts.

This paper addresses the problem of epitome construction for image compression. An optimized epitome construction method is first described, where the epitome and the associated image reconstruction, are both successively performed at full pel and sub-pel accuracy. The resulting complete still image compression scheme is then discussed with details on some innovative tools. The PSNR-rate performance achieved with this epitome-based compression method is significantly higher than the one obtained with H.264 Intra and with state of the art epitome construction method. A bit-rate saving up to 16% comparatively to H.264 Intra is achieved.

This paper addresses the issue of video compression within professional studio environments. In particular, the study concerns the degradation of the video quality in the case of successive encoding/decoding operations in the presence of pixel shifting to reflect some processing that may occur during the production workflow. The three codecs currently discussed within the professional community have been considered, namely JPEG2000, L264/MPEG-4 AVC and DIRAC and applied to HDTV 4:2:2 10 bits contents. The main contribution of the proposed paper is to provide inputs for professional video studios to select appropriate technologies for their business. By stressing the different encoders, it also underlines the limitations or difficulties some schemes may have compared to the others. Finally, some interesting perspectives are proposed to improve the requirements for possible future encoding schemes.

This paper presents an implementation of an end-to-end application for streaming stored MPEG-4 fine-grained scalable (FGS) videos over the best-effort Internet. Our current implementation runs over TCP but can also be run over TCP-friendly RTP/UDP with an error recovery mechanism. Our scheme adapts the coding rate of the streaming video to the variations of the available bandwidth for the connection, while smoothing changes in image quality between consecutive video scenes. We justify our design choices and present the results of the simulations obtained from our testbed. Our combined use of FGS-encoded video with a simple network-adaptation heuristic gives a system of low complexity, which is suitable for high performing video servers.