Gaze behavior of virtual characters in video games and virtual reality experiences is a key factor of realism and immersion. Indeed, gaze plays many roles when interacting with the environment; not only does it indicate what characters are looking at, but it also plays an important role in verbal and non-verbal behaviors and in making virtual characters alive. Automated computing of gaze behaviors is however a challenging problem, and to date none of the existing methods are capable of producing close-to-real results in an interactive context. We therefore propose a novel method that leverages recent advances in several distinct areas related to visual saliency, attention mechanisms, saccadic behavior modelling, and head-gaze animation techniques. Our approach articulates these advances to converge on a multi-map saliency-driven model which offers real-time realistic gaze behaviors for non-conversational characters, together with additional user-control over customizable features to compose a wide variety of results. We first evaluate the benefits of our approach through an objective evaluation that confronts our gaze simulation with ground truth data using an eye-tracking dataset specifically acquired for this purpose. We then rely on subjective evaluation to measure the level of realism of gaze animations generated by our method, in comparison with gaze animations captured from real actors. Our results show that our method generates gaze behaviors that cannot be distinguished from captured gaze animations. Overall, we believe that these results will open the way for more natural and intuitive design of realistic and coherent gaze animations for real-time applications.

Abstract Aesthetic quality assessment for photographs is an important research topic since it can be used for a number of applications, such as image database management or image browsing. In 2012, the aesthetic visual analysis (AVA) dataset has been proposed. It has since then been used to train the majority of computational models of aesthetics assessment. We observe that AVA is mainly composed of competitive photographs which notion of aesthetics differs from other kinds of photographs, such as professional photographs. In this paper, we evaluate whether or not recent aesthetics assessment models generalize well and perform well over professional photographs. We noticed that the different models we tested behave differently on both categories, and therefore do not generalize well. Besides, we fine‐tuned one of the tested model using professional photographs and the results show that this fine‐tuning is effectively improving the coverage of the methods.

Abstract Example‐based colour transfer between images, which has raised a lot of interest in the past decades, consists of transferring the colour of an image to another one. Many methods based on colour distributions have been proposed, and more recently, the efficiency of neural networks has been demonstrated again for colour transfer problems. In this paper, we propose a new pipeline with methods adapted from the image domain to automatically transfer the colour from a target point cloud to an input point cloud. These colour transfer methods are based on colour distributions and account for the geometry of the point clouds to produce a coherent result. The proposed methods rely on simple statistical analysis, are effective, and succeed in transferring the colour style from one point cloud to another. The qualitative results of the colour transfers are evaluated and compared with existing methods.

High Dynamic Range (HDR) and Wide Color Gamut (WCG) screens are able to display images with brighter and darker pixels with more vivid colors than ever. Automatically assessing the quality of these HDR/WCG images is of critical importance to evaluate the performances of image compression schemes. In recent years, full-reference metrics, such as HDR-VDP-2, PU-encoding metrics, have been designed for this purpose. However, none of these metrics consider chromatic artifacts. In this paper, we propose our own full-reference quality metric adapted to HDR and WCG content that is sensitive to chromatic distortions. The proposed metric is based on two existing HDR quality metrics and color image features. A support vector machine regression is used to combine the aforementioned features. Experimental results demonstrate the effectiveness of the proposed metric in the context of image compression.

Recent progress in digital photography and storage availability has drastically changed our approach to photo creation. While in the era of film cameras, careful forethought would usually precede the capture of a photo; nowadays, a large number of pictures can be taken with little effort. One of the consequences is the creation of numerous photos depicting the same moment in slightly different ways, which makes the process of organizing photos laborious for the photographer. Nevertheless, photo collection organization is important both for exploring photo albums and for simplifying the ultimate task of selecting the best photos. In this work, we conduct a user study to explore how users tend to organize or cluster similar photos in albums, to what extent different users agree in their clustering decisions, and to investigate how the clustering-defined photo context affects the subsequent photo-selection process. We also propose an automatic hierarchical clustering solution for modeling user clustering decisions. To demonstrate the usefulness of our approach, we apply it to the task of automatic photo evaluation within photo albums and propose a clustering-based context adaptation.

The selection of the best photos in personal albums is a task that is often faced by photographers. This task can become laborious when the photo collection is large and it contains multiple similar photos. Recent advances on image aesthetics and photo importance evaluation has led to the creation of different metrics for automatically assessing a given image. However, these metrics are intended for the independent assessment of an image, without considering the possible context implicitly present within photo albums. In this work, we perform a user study for assessing how users select photos when provided with a complete photo album---a task that better reflects how users may review their personal photos and collections. Using the data provided by our study, we evaluate how existing state-of-the-art photo assessment methods perform relative to user selection, focusing in particular on deep learning based approaches. Finally, we explore a recent framework for adapting independent image scores to collections and evaluate in which scenarios such an adaptation can prove beneficial.

Abstract The placement of lights in a 3D scene is a technical and artistic task that requires time and trained skills. Most 3D modelling tools only provide a direct control of light sources, through the manipulation of parameters such as size, location, flux (the perceived power of light) or opening angle (the light frustum). Approaches have been relying on automated or semi‐automated techniques to relieve users from such low‐level manipulations at the expense of an important computational cost. In this paper, guided by discussions with experts in scene and object lighting, we propose an indirect control of area light sources. We first formalize the classical 3‐point lighting design principle (key‐light, fill‐lights and back/rim‐lights) in a parametric model. Given a key‐light placed in the scene, we then provide a computational approach to (i) automatically compute the position and size of fill‐lights and back/rim‐lights by analyzing the geometry of 3D character, and (ii) automatically compute the flux and size of key, fill and back/rim lights, given a sample reference image in a computationally efficient way. Results demonstrate the benefits of the approach on the quick lighting of 3D characters, and further demonstrate the feasibility of interactive control of multiple lights through image features.

Multivariate generalized Gaussian distributions (MGGDs) have aroused a great interest in the image processing community thanks to their ability to describe accurately various image features, such as image gradient fields. However, so far their applicability has been limited by the lack of a transformation between two of these parametric distributions. In this paper, we propose a novel transformation between MGGDs, consisting of an optimal transportation of the second-order statistics and a stochastic-based shape parameter transformation. We employ the proposed transformation between MGGDs for a color transfer and a gradient transfer between images. We also propose a new simultaneous transfer of color and gradient, which we apply for image color correction.

In this paper, we propose a perceptual model for evaluating results from color transfer methods. We conduct a user study, which provides a set of subjective scores for triplets of input, target and result images. Then, for each triplet, we compute a number of image features, which objectively characterize a color transfer. To describe the relationship between these features and the subjective scores, we build a regression model with random forests. An analysis and a cross-validation show that the predictions of our model are highly accurate.

This paper proposes a brightness control method for retargeting legacy content on High Dynamic Range (HDR) displays. This method is based on a perceptual experiment that evaluates the brightness preference of users. We study the special case of screen content to better understand the effect of image statistics on human preference for brightness. The experiment concluded that brightness preference is highly content dependent. Using the data of the subjective study, we propose a new feature that analyses the source content to appropriately control the brightness of the HDR display. Based on this feature, we propose a brightness control algorithm that adjusts the luminance of display depending on the number of bright pixels in the content.

The human visual system is sensitive to relative differences in luminance, but light transport simulation algorithms based on Metropolis sampling often result in a highly nonuniform relative error distribution over the rendered image. Although this issue has previously been addressed in the context of the Metropolis light transport algorithm, our work focuses on Metropolis photon tracing. We present a new target function (TF) for Metropolis photon tracing that ensures good stratification of photons leading to pixel estimates with equalized relative error. We develop a hierarchical scheme for progressive construction of the TF from paths sampled during rendering. In addition to the approach taken in previous work, where the TF is defined in the image plane, ours can be associated with compact spatial regions. This allows us to take advantage of illumination coherence to more robustly estimate the TF while adapting to geometry discontinuities. To sample from this TF, we design a new replica exchange Metropolis scheme. We apply our algorithm in progressive photon mapping and show that it often outperforms alternative approaches in terms of image quality by a large margin.

The vast majority of video content existing today is in Standard Dynamic Range (SDR) format and there is a strong interest in upscaling this content for upcoming High Dynamic Range (HDR) displays. Tone expansion or inverse tone mapping converts SDR content into HDR format using Expansion Operators (EO). In this paper, we show that current EO’s do not perform as well when dealing with content of various lighting style aesthetics. In addition to this, we present a series of perceptual user studies evaluating user preference for lighting style in HDR content. This study shows that tone expansion of stylized content takes the form of gamma correction and we propose a method that adapts the gamma value to the style of the video. We validate our method through a subjective evaluation against state-of-the-art methods. Furthermore, our work has been oriented for 1000 nits HDR displays and we present a framework positioning our method in conformance with existing SDR standards and upcoming HDR TV standards.

We propose a fast Computer-Generated Hologram (CGH) computation method based on a hybrid point-source/wave-field approach. Whereas previously proposed methods tried to reduce the computational complexity of the point-source or the wave-field approaches independently, our method uses the two approaches together and therefore takes advantages from both of them. The algorithm consists of three steps. First, the 3D scene is sliced into several depth layers parallel to the hologram plane. Then, for each layer, we compute the complex wave scattered by this layer either using a wave-field or a point-source approach according to a threshold criterion on the number of points within the layer. Finally, we sum up the complex waves scattered by all the depth layers in order to obtain the final CGH. Experimental results reveal that this combination of approaches does not produce any visible artifact and outperforms both the point-source and wave-field approaches.

Tone Mapping Operators (TMOs) transform High Dynamic Range (HDR) contents to address Low Dynamic Range (LDR) displays. However, before reaching the end-user, these contents are usually compressed using a codec (coder-decoder) for broadcasting or storage purposes. Achieving the best trade-off between rendering and compression efficiency is of prime importance. Any TMO includes a rounding quantization to convert floating point values to integer ones. In this work, we propose to modify this quantization to increase the compression efficiency of the tone mapped content. By using a motion compensation, our technique preserves the rendering intent of the TMO while maximizing the correlations between successive frames. Experimental results show that we can save up to 12% of the total bit-rate as well as an average bit-rate reduction of 8.5% for all the test sequences. We show that our technique can be applied to other applications such as denoising.

Abstract Color adaptation is a well known ability of the human visual system (HVS). Colors are perceived as constant even though the illuminant color changes. Indeed, the perceived color of a diffuse white sheet of paper is still white even though it is illuminated by a single orange tungsten light, whereas it is orange from a physical point of view. Unfortunately global illumination algorithms only focus on the physics aspects of light transport. The ouput of a global illuminantion engine is an image which has to undergo chromatic adaptation to recover the color as perceived by the HVS. In this paper, we propose a new color adaptation method well suited to global illumination. This method estimates the adaptation color by averaging the irradiance color arriving at the eye. Unlike other existing methods, our approach is not limited to the view frustrum, as it considers the illumination from all the scene. Experiments have shown that our method outperforms the state of the art methods.

This paper studies the design and application of a novel visual attention model designed to compute user's gaze position automatically, i.e., without using a gaze-tracking system. The model we propose is specifically designed for real-time first-person exploration of 3D virtual environments. It is the first model adapted to this context which can compute in real time a continuous gaze point position instead of a set of 3D objects potentially observed by the user. To do so, contrary to previous models which use a mesh-based representation of visual objects, we introduce a representation based on surface-elements. Our model also simulates visual reflexes and the cognitive processes which take place in the brain such as the gaze behavior associated to first-person navigation in the virtual environment. Our visual attention model combines both bottom-up and top-down components to compute a continuous gaze point position on screen that hopefully matches the user's one. We conducted an experiment to study and compare the performance of our method with a state-of-the-art approach. Our results are found significantly better with sometimes more than 100 percent of accuracy gained. This suggests that computing a gaze point in a 3D virtual environment in real time is possible and is a valid approach, compared to object-based approaches. Finally, we expose different applications of our model when exploring virtual environments. We present different algorithms which can improve or adapt the visual feedback of virtual environments based on gaze information. We first propose a level-of-detail approach that heavily relies on multiple-texture sampling. We show that it is possible to use the gaze information of our visual attention model to increase visual quality where the user is looking, while maintaining a high-refresh rate. Second, we introduce the use of the visual attention model in three visual effects inspired by the human visual system namely: depth-of-field blur, camera- motions, and dynamic luminance. All these effects are computed based on the simulated gaze of the user, and are meant to improve user's sensations in future virtual reality applications.

This paper introduces a novel visual attention model to compute user's gaze position automatically, i.e. without using a gaze-tracking system. Our model is specifically designed for real-time first-person exploration of 3D virtual environments. It is the first model adapted to this context which can compute, in real-time, a continuous gaze point position instead of a set of 3D objects potentially observed by the user. To do so, contrary to previous models which use a mesh-based representation of visual objects, we introduce a representation based on surface-elements. Our model also simulates visual reflexes and the cognitive process which takes place in the brain such as the gaze behavior associated to first-person navigation in the virtual environment. Our visual attention model combines the bottom-up and top-down components to compute a continuous gaze point position on screen that hopefully matches the user's one. We have conducted an experiment to study and compare the performance of our method with a state-of-the-art approach. Our results are found significantly better with more than 100% of accuracy gained. This suggests that computing in realtime a gaze point in a 3D virtual environment is possible and is a valid approach as compared to object-based approaches.

Abstract This paper introduces the use of a visual attention model to improve the accuracy of gaze tracking systems. Visual attention models simulate the selective attention part of the human visual system. For instance, in a bottom‐up approach, a saliency map is defined for the image and gives an attention weight to every pixel of the image as a function of its colour, edge or intensity. Our algorithm uses an uncertainty window, defined by the gaze tracker accuracy, and located around the gaze point given by the tracker. Then, using a visual attention model, it searches for the most salient points, or objects, located inside this uncertainty window, and determines a novel, and hopefully, better gaze point. This combination of a gaze tracker together with a visual attention model is considered as the main contribution of the paper. We demonstrate the promising results of our method by presenting two experiments conducted in two different contexts: (1) a free exploration of a visually rich 3D virtual environment without a specific task, and (2) a video game based on gaze tracking involving a selection task. Our approach can be used to improve real‐time gaze tracking systems in many interactive 3D applications such as video games or virtual reality applications. The use of a visual attention model can be adapted to any gaze tracker and the visual attention model can also be adapted to the application in which it is used.

This paper describes the use of user's focus point to improve some visual effects in virtual environments (VE). First, we describe how to retrieve user's focus point in the 3D VE using an eye-tracking system. Then, we propose the adaptation of two rendering techniques which aim at improving users' sensations during first-person navigation in VE using his/her focus point: (1) a camera motion which simulates eyes movement when walking, i.e., corresponding to vestibulo-ocular and vestibulocollic reflexes when the eyes compensate body and head movements in order to maintain gaze on a specific target, and (2) a depth-of-field (DoF) blur effect which simulates the fact that humans perceive sharp objects only within some range of distances around the focal distance. Second, we describe the results of an experiment conducted to study users' subjective preferences concerning these visual effects during first-person navigation in VE. It showed that participants globally preferred the use of these effects when they are dynamically adapted to the focus point in the VE. Taken together, our results suggest that the use of visual effects exploiting users' focus point could be used in several VR applications involving first- person navigation such as the visit of architectural site, training simulations, video games, etc.

This paper studies the use of visual blur effects, i.e., blurring of parts of the image fed back to the user, for First-Person-Navigations in Virtual Environments (VE). First, we introduce a model of dynamic visual blur for VE which is based on two types of blur effect: (1) a Depth-of-Field blur (DoF blur) which simulates the blurring of objects located in front or back of the focus point of the eyes, and (2) a peripheral blur which simulates the blurring of objects located at the periphery of the field of vision. We introduce two new techniques to improve real-time DoF: (1) a paradigm to compute automatically the focal distance, and (2) a temporal filtering that simulates the accommodation phenomenon. Second, we describe the results of a pilot experiment conducted to study the influence of blur effects on the performance and preference of video gamers during multiplayer sessions. Interestingly, it seems that visual blur effects did not degrade performance of gamers and they were preferred and selected by nearly half of the participants to improve fun and game-play. Taken together, our results suggest that the use of visual blur effects could thus be suitable in videogames and in other virtual environments.