High dynamic range (HDR) images require tone-mapping to be viewed on low dynamic range (LDR) displays. The performance of tone-mapping algorithms can be evaluated through a subjective study in which participants based on their liking rank or score tone-mapped images (TMIs). Subjective evaluation can be painstakingly slow; therefore, several quantitative metrics have been proposed for objective evaluation. This article presents a new robust metric that uses 16 features, measuring the loss of color, contrast, brightness, and structure, extracted from the test TMI and the reference HDR image. The effect of these attributes on image quality is investigated and combined into a single score in the [0, 1] range describing the quality of TMI. We validate the performance of the proposed metric by comparing it with 24 existing state-of-the-art metrics. The study uses two subjective datasets of TMIs, including one existing benchmark dataset and a new proposed dataset comprising HDR images of a variety of scenes, and a dataset of traditional images not generated through tone-mapping. In these studies, our method shows the highest correlation with subjective scores for both datasets of TMIs and remains in the second position for the dataset of traditional images.

Subjective evaluation of tone-mapped images is tedious and time-consuming; therefore, it is desirable to have algorithms for automatic quality assessment. Many full-reference and blind metrics have been developed for this purpose, but their performance is generally evaluated on limited benchmark datasets. This leaves a possibility that the observed performance of the metric could be due to overfitting, and it might indeed not perform well for all scenes. In this work, we propose a novel framework using population-based metaheuristics to evaluate the performance of these metrics without requiring any subjectively evaluated reference dataset. The proposed algorithm does not modify the individual image pixels, instead, the tone-mapping curve is modified to synthesize realistic tone-mapped images for evaluation. Moreover, it is not required to know the underlying model of the evaluated metric, which is treated just like a black box and can be replaced by any other metric seamlessly. Therefore, any new metrics designed in the future can also be easily evaluated by simply replacing just one module in the proposed evaluation framework. We evaluate six existing metrics and synthesize images to which the metrics fail to assign appropriate scores for visual quality. We also propose a method to rank the relative performance of evaluated metrics, through a competition in which each metric tries to find the errors in the scores given by other metrics.

Quality assessment of tone-mapped images has been an active research area over the past years. Humans have a personal liking of image attributes such as vividness of colors, mean brightness level, and contrast. Therefore, image quality is often determined through subjective studies. The participants are asked to assign scores to the images, and the average scores are used to determine the relative quality of the images or the algorithms that produced them. Several metrics have been proposed that try to replicate this process and assign quantitative scores to the images. While these metrics perform reasonably in general and their scores correlate well with subjective scores, they can fail badly in some situations. This paper proposes a novel method to put these metrics to test under extreme conditions and observe their performance. For this, we use the differential evolution approach that keeps modifying the tone-mapping function iteratively, such that the assigned score by the metric keeps increasing over iterations. We show that visible distortions start showing up in the image in many situations, yet the score given by the metric remains very high.