In an exploratory study, we tested the effects of two kinds of VR presentations of constellation drawings and labels on memorization. We asked participants to memorize two sets of 14 constellations each: one set presented all at once, and another presented one by one. We evaluated their memorization skills by immediately conducting a test after each memorization set, removing half of the labels and half of the drawings for participants to name or draw, respectively. The results of this within-subjects pilot study point towards the concurrent (show-all) presentation of stimuli having higher scores than the individual (one-by-one) condition. However, the selection of the constellations may have influenced the preliminary results, hence, further modification of the evaluation procedure is required for future work.

Image restoration (IR) via deep learning has been vigorously studied in recent years. However, due to the ill-posed nature of the problem, it is challenging to recover the high-quality image details from a single distorted input especially when images are corrupted by multiple distortions. In this paper, we propose a multi-stage IR approach for progressive restoration of multi-degraded images via transferring similar edges/textures from the reference image. Our method, called a Reference-based Image Restoration Transformer (Ref-IRT), operates via three main stages. In the first stage, a cascaded U-Transformer network is employed to perform the preliminary recovery of the image. The proposed network consists of two U-Transformer architectures connected by feature fusion of the encoders and decoders, and the residual image is estimated by each U-Transformer in an easy-to-hard and coarse-to-fine fashion to gradually recover the high-quality image. The second and third stages perform texture transfer from a reference image to the preliminarily-recovered target image to further enhance the restoration performance. To this end, a quality-degradation-restoration method is proposed for more accurate content/texture matching between the reference and target images, and a texture transfer/reconstruction network is employed to map the transferred features to the high-quality image. Experimental results tested on three benchmark datasets demonstrate the effectiveness of our model as compared with other state-of-the-art multi-degraded IR methods. Our code and dataset are available at https://vinelab.jp/refmdir/.

Previous research on no-reference (NR) quality assessment of multiply-distorted images focused mainly on three distortion types (white noise, Gaussian blur, and JPEG compression), while in practice images can be contaminated by many other common distortions due to the various stages of processing. Although MUSIQUE (MUltiply-and Singly-distorted Image QUality Estimator) Zhang et al., TIP 2018 is a successful NR algorithm, this approach is still limited to the three distortion types. In this paper, we extend MUSIQUE to MUSIQUE-II to blindly assess the quality of images corrupted by five distortion types (white noise, Gaussian blur, JPEG compression, JPEG2000 compression, and contrast change) and their combinations. The proposed MUSIQUE-II algorithm builds upon the classification and parameter-estimation framework of its predecessor by using more advanced models and a more comprehensive set of distortion-sensitive features. Specifically, MUSIQUE-II relies on a three-layer classification model to identify 19 distortion types. To predict the five distortion parameter values, MUSIQUE-II extracts an additional 14 contrast features and employs a multi-layer probability-weighting rule. Finally, MUSIQUE-II employs a new most-apparent-distortion strategy to adaptively combine five quality scores based on outputs of three classification models. Experimental results tested on three multiply-distorted and six singly-distorted image quality databases show that MUSIQUE-II yields not only a substantial improvement in quality predictive performance as compared with its predecessor, but also highly competitive performance relative to other state-of-the-art FR/NR IQA algorithms.

Visual saliency detection is a useful technique for predicting, which regions humans will tend to gaze upon in any given image. Over the last several decades, numerous algorithms for automatic saliency detection have been proposed and shown to work well on both synthetic and natural images. However, two key challenges remain largely unaddressed: 1) How to improve the relatively low predictive performance for images that contain large objects and 2) how to perform saliency detection on a wider variety of images from various categories without training. In this paper, we propose a new saliency detection algorithm that addresses these challenges. Our model first detects potentially salient regions based on multiscale extrema of local perceived color differences measured in the CIELAB color space. These extrema are highly effective for estimating the locations, sizes, and saliency levels of candidate regions. The local saliency candidates are further refined via two global extrema-based features, and then a Gaussian mixture is used to generate the final saliency map. Experimental validation on the extensive CAT2000 data set demonstrates that our proposed method either outperforms or is highly competitive with prior approaches, and can perform well across different categories and object sizes, while remaining training-free.

Over the past couple of decades, numerous image quality assessment (IQA) algorithms have been developed to estimate the quality of images that contain a single type of distortion. Although in practice, images can be contaminated by multiple distortions, previous research on quality assessment of multiply-distorted images is very limited. In this paper, we propose an efficient algorithm to blindly assess the quality of both multiply and singly distorted images based on predicting the distortion parameters using a bag of natural scene statistics (NSS) features. Our method, called MUltiply- and Singlydistorted Image QUality Estimator (MUSIQUE), operates via three main stages. In the first stage, a two-layer classification model is employed to identify the distortion types (i.e., Gaussian blur, JPEG compression, and white noise) that may exist in an image. In the second stage, specific regression models are employed to predict the three distortion parameters (i.e., σG for Gaussian blur, Q for JPEG compression, and σN for white noise) by learning the different NSS features for different distortion types and combinations. In the final stage, the three estimated distortion parameter values are mapped and combined into an overall quality estimate based on quality-mapping curves and the most-apparent-distortion strategy. Experimental results tested on three multiply-distorted and seven singly-distorted image quality databases demonstrate that the proposed MUSIQUE algorithm can achieve better/competitive performance as compared to other state-of-the-art FR/NR IQA algorithms.

The recent popularity of remote desktop software and live streaming of composited video has given rise to a growing number of applications which make use of so-called screen content images that contain a mixture of text, graphics, and photographic imagery. Automatic quality assessment (QA) of screen-content images is necessary to enable tasks such as quality monitoring, parameter adaptation, and other optimizations. Although QA of natural images has been heavily researched over the last several decades, QA of screen content images is a relatively new topic. In this paper, we present a QA algorithm, called convolutional neural network (CNN) based screen content image quality estimator (CNN-SQE), which operates via a fuzzy classification of screen content images into plain-text, computergraphics/ cartoons, and natural-image regions. The first two classes are considered to contain synthetic content (text/graphics), and the latter two classes are considered to contain naturalistic content (graphics/photographs), where the overlap of the classes allows the computer graphics/cartoons segments to be analyzed by both text-based and natural-image-based features. We present a CNN-based approach for the classification, an edge-structurebased quality degradation model, and a region-size-adaptive quality-fusion strategy. As we will demonstrate, the proposed CNN-SQE algorithm can achieve better/competitive performance as compared with other state-of-the-art QA algorithms.

Algorithms for no-reference (NR) stereoscopic image quality assessment (SIQA) aim to evaluate the perceptual quality of a stereoscopic/3D image without the assistance of its reference. Current NR SIQA models often require training on 3D distorted images and their associated human opinion scores, which ultimately restrict their further application. In this paper, we present a simple yet effective NR SIQA model that does not require training on existing 3D image databases. Instead, we train our model on a large dataset of natural stereoscopic images based on learning the local statistics of the Cyclopean contrast maps, and then use the existing 2D NR IQA model to help guide the NR SIQA task. Experimental results demonstrate the efficacy of our proposed method.

A crucial requirement for modern image coding is the ability to accurately and efficiently predict the local visibility of coding artifacts. Such predictions could help guide the allocation of bits or the determination of quality for each spatial region. This paper presents a convolutional-neural-network-based (CNN-based) model to predict local distortion visibility in natural scenes. Although CNNs have recently emerged as a powerful tool for many computer vision applications due to its deep learning abilities and computational efficiency, CNNs have never been tested for predicting continuous values such as visibility thresholds. We optimized the model's parameters on our recently published large dataset on local masking in natural scenes [Alam et al., Journal of Vision, 2014]. Testing results demonstrate that our CNN-based model: (1) can indeed succeed in this task; (2) can more accurately predict thresholds than modern gain-control-based models; (3) is competitive in terms of prediction accuracy with a gain-control model tuned to the same dataset; and (4) is significantly more computationally efficient than modern gain-controls models.

Grayscale logo watermarking is a quite well-developed area of digital image watermarking which seeks to embed into the host image another smaller logo image. The key advantage of such an approach is the ability to visually analyze the extracted logo for rapid visual authentication and other visual tasks. However, logos pose new challenges for invisible watermarking applications which need to keep the watermark imperceptible within the host image while simultaneously maintaining robustness to attacks. This paper presents an algorithm for invisible grayscale logo watermarking that operates via adaptive texturization of the logo. The central idea of our approach is to recast the watermarking task into a texture similarity task. We first separate the host image into sufficiently textured and poorly textured regions. Next, for textured regions, we transform the logo into a visually similar texture via the Arnold transform and one lossless rotation; whereas for poorly textured regions, we use only a lossless rotation. The iteration for the Arnold transform and the angle of lossless rotation are determined by a model of visual texture similarity. Finally, for each region, we embed the transformed logo into that region via a standard wavelet-based embedding scheme. We employ a multistep extraction stage, in which an affine parameter estimation is first performed to compensate for possible geometrical transformations. Testing with multiple logos on a database of host images and under a variety of attacks demonstrates that the proposed algorithm yields better overall performance than competing methods.

Algorithms for a stereoscopic image quality assessment (IQA) aim to estimate the qualities of 3D images in a manner that agrees with human judgments. The modern stereoscopic IQA algorithms often apply 2D IQA algorithms on stereoscopic views, disparity maps, and/or cyclopean images, to yield an overall quality estimate based on the properties of the human visual system. This paper presents an extension of our previous 2D most apparent distortion (MAD) algorithm to a 3D version (3D-MAD) to evaluate 3D image quality. The 3D-MAD operates via two main stages, which estimate perceived quality degradation due to 1) distortion of the monocular views and 2) distortion of the cyclopean view. In the first stage, the conventional MAD algorithm is applied on the two monocular views, and then the combined binocular quality is estimated via a weighted sum of the two estimates, where the weights are determined based on a block-based contrast measure. In the second stage, intermediate maps corresponding to the lightness distance and the pixel-based contrast are generated based on a multipathway contrast gain-control model. Then, the cyclopean view quality is estimated by measuring the statistical-difference-based features obtained from the reference stereopair and the distorted stereopair, respectively. Finally, the estimates obtained from the two stages are combined to yield an overall quality score of the stereoscopic image. Tests on various 3D image quality databases demonstrate that our algorithm significantly improves upon many other state-of-the-art 2D/3D IQA algorithms.

This paper presents an algorithm for detecting multiple salient objects in images. The algorithm extends our previous algorithm which was designed to detect only a single salient object. The new algorithm employs five feature maps (lightness distance, color distance, contrast, sharpness, and edge strength), along with a new image-adaptive technique for estimating the usefulness of each feature map based on a local measure of cluster density. As we will demonstrate, our new version can successfully detect multiple salient objects on images for which the previous version did not succeed. Testing on subsets of images from two databases shows that the proposed algorithm performs well on a variety of images containing multiple salient objects.

In this letter, we present a simple, yet effective wavelet-based algorithm for estimating both global and local image sharpness (FISH, Fast Image Sharpness). FISH operates by first decomposing the input image via a three-level separable discrete wavelet transform (DWT). Next, the log-energies of the DWT subbands are computed. Finally, a scalar index corresponding to the image's overall sharpness is computed via a weighted average of these log-energies. Testing on several image databases demonstrates that, despite its simplicity, FISH is competitive with the currently best-performing techniques both for sharpness estimation and for no-reference image quality assessment.

This paper presents an algorithm designed to measure the local perceived sharpness in an image. Our method utilizes both spectral and spatial properties of the image: For each block, we measure the slope of the magnitude spectrum and the total spatial variation. These measures are then adjusted to account for visual perception, and then, the adjusted measures are combined via a weighted geometric mean. The resulting measure, i.e., S(3) (spectral and spatial sharpness), yields a perceived sharpness map in which greater values denote perceptually sharper regions. This map can be collapsed into a single index, which quantifies the overall perceived sharpness of the whole image. We demonstrate the utility of the S(3) measure for within-image and across-image sharpness prediction, no-reference image quality assessment of blurred images, and monotonic estimation of the standard deviation of the impulse response used in Gaussian blurring. We further evaluate the accuracy of S(3) in local sharpness estimation by comparing S(3) maps to sharpness maps generated by human subjects. We show that S(3) can generate sharpness maps, which are highly correlated with the human-subject maps.

This paper presents an algorithm for video quality assessment, spatiotemporal MAD (ST-MAD), which extends our previous image-based algorithm (MAD [1]) to take into account visual perception of motion artifacts. ST-MAD employs spatiotemporal “images” (STS images [2]) created by taking time-based slices of the original and distorted videos. Motion artifacts manifest in the STS images as spatial artifacts, which allows one to quantify motion-based distortion by using classical image-quality assessment techniques. ST-MAD estimates motion-based distortion by applying MAD's appearance-based model to compare the distorted video's STS images to the original video's STS images. This comparison is further adjusted by using optical-flow-derived weights designed to give greater precedence to fast-moving regions located toward the center of the video. Testing on the LIVE video database demonstrates that ST-MAD performs well in predicting video quality.

In this paper we present an algorithm which uses adaptive selection of low-level features for main subject detection. The algorithm first computes low-level features such as contrast and sharpness, each computed in a block-based fashion. Next, the algorithm quantifies the usefulness of each feature by using both statistical and geometric information measured across blocks. Finally, the saliency of each block is determined via a weighted linear combination of the features, where the weights are chosen based on each feature's estimated usefulness. Our results demonstrate that the adaptive nature of this algorithm allows it to perform competitively with other techniques, while maintaining very low computational complexity.

This paper presents the results of a computational experiment designed to investigate the extent to which metrics of image fidelity can be improved through knowledge of where humans tend to fixate in images. Five common metrics of image fidelity were augmented using two sets of fixation data, one set obtained under task-free viewing conditions and another set obtained when viewers were asked to judge image quality. The augmented metrics were then compared to subjective ratings of the images. The results show that most metrics can be improved using eye fixation data, but a greater improvement was found using fixations obtained in the task-free condition (task-free viewing).

This paper presents an algorithm designed to compute the perceived interest of objects in images. We measured likelihood functions via a psychophysical experiment in which subjects rated the perceived visual interest of 562 objects in 150 images. These results were then used to determine the likelihood of perceived interest given various factors such as location, contrast, color, and edge-strength. These likelihood functions are used as part of a Bayesian formulation in which perceived interest is inferred based on the factors. Our results demonstrate that our algorithm can perform well in predicting perceived interest.

This paper presents an efficient metric for quantifying the visual fidelity of natural images based on near-threshold and suprathreshold properties of human vision. The proposed metric, the visual signal-to-noise ratio (VSNR), operates via a two-stage approach. In the first stage, contrast thresholds for detection of distortions in the presence of natural images are computed via wavelet-based models of visual masking and visual summation in order to determine whether the distortions in the distorted image are visible. If the distortions are below the threshold of detection, the distorted image is deemed to be of perfect visual fidelity (VSNR = infinity) and no further analysis is required. If the distortions are suprathreshold, a second stage is applied which operates based on the low-level visual property of perceived contrast, and the mid-level visual property of global precedence. These two properties are modeled as Euclidean distances in distortion-contrast space of a multiscale wavelet decomposition, and VSNR is computed based on a simple linear sum of these distances. The proposed VSNR metric is generally competitive with current metrics of visual fidelity; it is efficient both in terms of its low computational complexity and in terms of its low memory requirements; and it operates based on physical luminances and visual angle (rather than on digital pixel values and pixel-based dimensions) to accommodate different viewing conditions.

Wavelet-based spatial quantization is a technique to compress image data that adapts the compression to the data in each region of an image. This approach is motivated because quantization with a single step-size does not result in a uniform visual effect across each spatial location; different types of image content mask quantization errors in different ways. While many spatial quantization techniques determine step-sizes via local activity measures, the proposed method induces local quantization distortion based on experiments that quantify human detection of this distortion as function of both the contrast and the type of the image data. Three types in particular, textures, structures and edges, are considered. A classifier is utilized to detect to which of these three categories a local region of image data belongs, and step-sizes are then derived based on the contrast and class of each region. Class and contrast data are conveyed to the coder with explicit side information. For images compressed at threshold, the proposed method requires 3-10 % less rate than a similar previous approach without classification, and on average produces images that are preferred by 2/3 of tested viewers.

Recent developments in psychovisual modeling have led to improvements in wavelet-based coder performance. A spatially selective quantizer based on texture masking sensitivities is introduced, which hides distortion in high-contrast portions of images. Unlike other spatial quantization schemes, this method requires explicit side information to convey stepsizes. A simple coder is presented which leverages this side information to reduce the rate required to code the quantized data. Side information coding is also discussed. With respect to visual quality, this compression scheme performs competitively with a CSF-optimized JPEG-2000 coder at equivalent rates.

This paper presents a contrast-based quantization strategy for use in lossy wavelet image compression that attempts to preserve visual quality at any bit rate. Based on the results of recent psychophysical experiments using near-threshold and suprathreshold wavelet subband quantization distortions presented against natural-image backgrounds, subbands are quantized such that the distortions in the reconstructed image exhibit root-mean-squared contrasts selected based on image, subband, and display characteristics and on a measure of total visual distortion so as to preserve the visual system's ability to integrate edge structure across scale space. Within a single, unified framework, the proposed contrast-based strategy yields images which are competitive in visual quality with results from current visually lossless approaches at high bit rates and which demonstrate improved visual quality over current visually lossy approaches at low bit rates. This strategy operates in the context of both nonembedded and embedded quantization, the latter of which yields a highly scalable codestream which attempts to maintain visual quality at all bit rates; a specific application of the proposed algorithm to JPEG-2000 is presented.

A visually-optimal quantization and rate-control strategy based on results of recent contrast sensitivity and suprathreshold summation experiments is proposed. At suprathreshold contrasts, masked detection thresholds for wavelet subband quantization distortions were approximately equal for scale-3, 4, and 5 distortions; approximately 52% greater for scale-2 distortions; and approximately 84% greater for scale-1 distortions. Based on a suprathreshold error-pooling model, contrasts for individual subbands are selected to match these contrast ratios, and are adjusted to account for changes in relative sensitivity at suprathreshold contrasts. Quantization step sizes are then computed from the adjusted base contrasts. A target contrast is estimated from the desired rate, and rate control is performed by adjusting this contrast until the rate is met. Images compressed with the proposed algorithm show improved visual quality at low bit rates.