The perception of audiovisual congruence arises from the brain’s integration of correspondent acoustic and visual cues in an environment. While individuals anticipate certain auditory qualities when exposed to specific visual cues, the perceptual tolerances governing these expectations remain largely unexplored. Understanding these tolerances is essential for designing plausible immersive virtual environments. We developed an immersive audiovisual presentation system featuring a 9-foot radius, 210-degree cylindrical screen and variable room acoustics to systematically investigate the perception of cross-modal sensory cue congruence. Using a two-interval forced choice paradigm, we measured subjective reports of relative audiovisual congruence across 16 virtual room configurations combining four visual room sizes with four reverberation durations. Six participants completed 120 paired comparisons each. We employed the Bradley-Terry-Luce model to derive a perceptual ranking of configurations. Our findings demonstrate that individuals reliably detect variations across a spectrum of congruence: configurations matching physical expectations (large rooms with long reverberation, small rooms with short reverberation) were perceived as most congruent, while mismatched configurations were perceived as least congruent. Intermediate configurations revealed specific sensitivities that inform tolerance boundaries. These results establish a framework for defining perceptual tolerances applicable to immersive virtual world design, where optimizing audiovisual plausibility can enhance user experience using defined limits of physically accurate simulation.

Estimating user quality of experience (QoE) for audiovisual media is essential for service providers, but it is generally infeasible to monitor self-reporting within consumer applications. We examine how eye-tracking data correlates with viewers’ subjective quality of experience during audiovisual media consumption. Using measured fixation sequences, we implement a foveated visible differences predictor model (FovVideoVDP) to quantify just-objectionable-differences (JODs). Subjective QoEs correlated with fixation rate, potentially indicating increased engagement with higher bitrates. Foveated JODs aligned closely with mean opinion scores, comparing favorably to the most commonly used video quality metrics.

The perceived discrepancy between continuous motion as seen in nature and frame-by-frame exhibition on a display, sometimes termed judder, is an integral part of video presentation. Over time, content creators have developed a set of rules and guidelines for maintaining a desirable cinematic look under the restrictions placed by display technology without incurring prohibitive judder. With the advent of novel displays capable of high brightness, contrast, and frame rates, these guidelines are no longer sufficient to present audiences with a uniform viewing experience. In this work, we analyze the main factors for perceptual motion artifacts in digital presentation and gather psychophysical data to generate a model of judder perception. Our model enables applications like matching perceived motion artifacts to a traditionally desirable level and maintain a cinematic motion look.

This study builds on previous work exploring machine learning and perceptual transforms in predicting overall display quality as a function of image quality dimensions that correspond to physical display parameters. Previously, we found that the use of perceptually transformed parameters or machine learning exceeded the performance of predictors using just physical parameters and linear regression. Further, the combination of perceptually transformed parameters with machine learning allowed for robustness to parameters outside of the data set, both for cases of interpolation and extrapolation. Here we apply machine learning at a more intrinsic level. We first evaluate how well the machine learning can develop predictors of the individual dimensions of the overall quality, and then how well those individual predictors can be consolidated across themselves to predict the overall display quality. Having predictions of individual dimensions of quality that are closely related to specific hardware design choices enables more nimble cost trade-off design options.

We present a study into the perception of display brightness as related to the physical size and distance of the screen from the observer. Brightness perception is a complex topic, which is influenced by a number of lower- and higher-order factors—with empirical evidence from the cinema industry suggesting that display size may play a significant role. To test this hypothesis, we conducted a series of user studies exploring brightness perception for a range of displays and distances from the observer that span representative use scenarios. Our results suggest that retinal size is not sufficient to explain the range of discovered brightness variations, but is sufficient in combination with physical distance from the observer. The resulting model can be used as a step toward perceptually correcting image brightness perception based on target display parameters. This can be leveraged for energy management and the preservation of artistic intent. A pilot study suggests that adaptation luminance is an additional factor for the magnitude of the effect.

High dynamic range imaging is currently being introduced to television, cinema and computer games. While it has been found that a fixed encoding for high dynamic range imagery needs at least 11 to 12 bits of tonal resolution, current mainstream image transmission interfaces, codecs and file formats are limited to 10 bits. To be able to use current generation imaging pipelines, this paper presents a baseband quantization scheme that exploits content characteristics to reduce the needed tonal resolution per image. The method is of low computational complexity and provides robust performance on a wide range of content types in different viewing environments and applications.

Color coding efficacy is critical to the success of any image compression method. The initial quantization step that happens during encoding limits the accuracy and directly affects compression performance. If too many code words are spent in areas of the color space that make little difference to perception, compression is taxed. If other areas are under-sampled, contour (banding) and color shifting artifacts can result. Legacy color encodings such as 24-bit sRGB do not easily extend to wide color gamuts and high dynamic range due to perceptual non-uniformity issues. In this paper, we compare a number of different color encodings and bit depths for suitability in HDR and wide gamut applications using a new metric for evaluating encoding efficacy, Number of Distinguishable Colors.

In this paper we present a novel perceptually-based algorithm for color quantization that produces images that consume less energy than conventionally quantized images when displayed on modern energy-adaptive displays. To evaluate the performance of the proposed algorithm, we performed a subjective study on a standard Kodak color image database. Experimental results indicate that the proposed algorithm is able to reduce the energy consumption by 4.25% on average, while achieving the same or better subjective image quality as conventional color quantization.

Recently, visual representations using high dynamic range (HDR) images become increasingly popular, with advancement of technologies for increasing the dynamic range of image. HDR image is expected to be used in wide-ranging applications such as digital cinema, digital photography and next generation broadcast, because of its high quality and its powerful expression ability. HDR imaging technologies will spread its sphere of influence in imaging industry. In this paper, we review the state-of-the-art studies and the trends of the HDR imaging, in terms of the following three points: (1) HDR imaging sensor and HDR image generation techniques as image acquisition technologies, (2) encode method of HDR images for efficient transmission and storage, (3) human visual system issues associated with reproduction of HDR image.

We propose a tone mapping operator that can minimize visible contrast distortions for a range of output devices, ranging from e-paper to HDR displays. The operator weights contrast distortions according to their visibility predicted by the model of the human visual system. The distortions are minimized given a display model that enforces constraints on the solution. We show that the problem can be solved very efficiently by employing higher order image statistics and quadratic programming. Our tone mapping technique can adjust image or video content for optimum contrast visibility taking into account ambient illumination and display characteristics. We discuss the differences between our method and previous approaches to the tone mapping problem.

Compared to cathode-ray tube (CRT) devices, the current liquid crystal display (LCD) devices have advantage on many aspects but disadvantage on displaying motion contents. LCD motion blur is caused jointly by the slow LCD's temporal response and the hold-type LCD's rendering method together with the motion pursuing function of human visual system (HVS). Although LCD motion blur has been addressed extensively in the literature, there is no systematic analysis. In this paper, a general LCD motion blur model, based on the sampling and reconstruction theory, is developed. This model, covering LCD and HVS, provides a systematic tool for quantitatively analyzing the LCD motion blur problem associated with any current and emerging LCD rendering technologies.

Luminance resolution of colour matrix displays can be enhanced when subpixel resampling is used to render images, but at the cost of colour aliasing if appropriate filtering is not done. The problem has been effectively solved for the case of one-dimensional RGB striped matrix displays by using a high-pass chrominance filter, based on a human perceptual model, following resampling. In this paper we motivate the case of 2D subpixel geometries and suggest an extension to the ID solution.

Subsampling based on subpixel addressing has been successfully applied to the problem of font rendering on an LCD. The increased horizontal resolution of the process makes otherwise fuzzy or aliased text look noticeably clearer, but the increased luminance resolution comes at the cost of chrominance aliasing. Also, the technique has only been applied to achromatic imagery. A new algorithm is presented which, based on a human perceptual model, successfully removes visible chrominance aliasing while retaining the invisible aliased components that contribute to enhanced resolution. The algorithm is applicable to both achromatic and natural colour imagery.

One of the common artifacts for three-row charge-coupled device (CCD) desktop scanners is color misregistration between the red, green, and blue layers of an image. This causes both color fringing and blur in the resulting scanned images, which we quantify by linear system theory analysis. Knowing the bandwidth and peak sensitivity asymmetries in the opponent color representation of the visual system, we develop a method to reduce the color misregistration artifact by attempting to capture signals in an approximate opponent color space. To facilitate separate capture of the luminance and chrominance signals, we use a new sensor arrangement. The luminance signal (Y) is captured at the full resolution using one row of the three-row CCD linear arrays. The first chrominance signal is captured on another row with a interleaved half resolution red (R) and half resolution luminance sensor elements and the second chrominance signal is similarly captured on a third row using blue (B) and luminance. Since each luminance and chrominance signal is isolated on a single row and since there is no registration error within a row, color misregistration is theoretically prevented in luminance as well as in the chrominance signals. Simulation shows that the new method does reduce blur and the visibility of color fringing. Because residual luminance and chrominance misregistration may occur we conduct a psychophysical experiment to judge the improvement in the scanned image quality. The experiment shows that this new capture scheme can significantly reduce the perception of misregistration artifacts. Finally, we use an image processing model of the visual system to quantify the visible differences due to misregistration and compare these to the psychophysical results.

One common visual optimization strategy for image compression is to exploit the visual masking effect where artifacts are locally masked by the image acting as a background signal. In this paper, we present a point-wise extended visual masking approach that nonlinearly maps the wavelet coefficients to a perceptually uniform domain prior to quantization by taking advantages of both self-contrast masking and neighborhood masking effects, thus achieving very good visual quality. It is essentially a coefficient-wise adaptive quantization without any overhead. It allows bitstream scalability, as opposed to many previous works. The proposed scheme has been adopted into the working draft of JPEG-2000 Part II.

We review the various tools in JPEG 2000 that allow the users to take advantages of the various properties of the human visual system such as spatial frequency sensitivity and the visual masking effect. We show that the visual tool sets in JPEG 2000 are much richer than what was available in JPEG, where only locally invariant frequency weighting can be exploited.

The paper describes an algorithm for the assessment of image fidelity. The algorithm includes an image processing model of the human visual system for luminance still imagery. The major components of the algorithm are described that model the visual system as three main sensitivity variations. These address the sensitivity as a function of gray level, as a function of spatial frequency, and as a function of image content. To quantify the performance of the algorithm, specific psychophysical experiments were simulated, and these results are shown.>

The principle involved in the uniform quantization of perceived lightness is considered in the context of digital radiography. Its effects on the visibility of quantization noise (contouring artifacts) are investigated both analytically and experimentally. Several visual system models of perceived lightness as a function of stimulus luminance are used. These include a logarithmic model, various power function models, and cone models. The results can be extended for other digital imaging modalities.