Image restoration (IR) via deep learning has been vigorously studied in recent years. However, due to the ill-posed nature of the problem, it is challenging to recover the high-quality image details from a single distorted input especially when images are corrupted by multiple distortions. In this paper, we propose a multi-stage IR approach for progressive restoration of multi-degraded images via transferring similar edges/textures from the reference image. Our method, called a Reference-based Image Restoration Transformer (Ref-IRT), operates via three main stages. In the first stage, a cascaded U-Transformer network is employed to perform the preliminary recovery of the image. The proposed network consists of two U-Transformer architectures connected by feature fusion of the encoders and decoders, and the residual image is estimated by each U-Transformer in an easy-to-hard and coarse-to-fine fashion to gradually recover the high-quality image. The second and third stages perform texture transfer from a reference image to the preliminarily-recovered target image to further enhance the restoration performance. To this end, a quality-degradation-restoration method is proposed for more accurate content/texture matching between the reference and target images, and a texture transfer/reconstruction network is employed to map the transferred features to the high-quality image. Experimental results tested on three benchmark datasets demonstrate the effectiveness of our model as compared with other state-of-the-art multi-degraded IR methods. Our code and dataset are available at https://vinelab.jp/refmdir/.

Accurate road centerline extraction is very important for many vital applications. In the road extraction, the acquisition of labeled data is time-consuming; thus, there is only a small amount of labeled samples in reality. To solve the problem of limited labeled samples, a semi-supervised road centerline extraction is proposed, which incorporates high-level feature selection, Markov random field (MRF), and ridge transversal method. The proposed road extraction approach consists of three steps: multiple features extraction, semi-supervised road area extraction, and road centerlines extraction. To get more abstract and discriminative high-level features, we apply multiple-feature adaptive sparse representation in mid-level features in different views generated by different prototype sets. To obtain an accurate road area result, we combine the feature learning framework with MRF. Then, we integrate Gabor filters and nonmaxima suppression with the ridge transversal method to extract centerlines. It is verified the proposed method achieves comparable performance with the state-of-the-art methods in terms of visual and quantitative aspects.

Previous research on no-reference (NR) quality assessment of multiply-distorted images focused mainly on three distortion types (white noise, Gaussian blur, and JPEG compression), while in practice images can be contaminated by many other common distortions due to the various stages of processing. Although MUSIQUE (MUltiply-and Singly-distorted Image QUality Estimator) Zhang et al., TIP 2018 is a successful NR algorithm, this approach is still limited to the three distortion types. In this paper, we extend MUSIQUE to MUSIQUE-II to blindly assess the quality of images corrupted by five distortion types (white noise, Gaussian blur, JPEG compression, JPEG2000 compression, and contrast change) and their combinations. The proposed MUSIQUE-II algorithm builds upon the classification and parameter-estimation framework of its predecessor by using more advanced models and a more comprehensive set of distortion-sensitive features. Specifically, MUSIQUE-II relies on a three-layer classification model to identify 19 distortion types. To predict the five distortion parameter values, MUSIQUE-II extracts an additional 14 contrast features and employs a multi-layer probability-weighting rule. Finally, MUSIQUE-II employs a new most-apparent-distortion strategy to adaptively combine five quality scores based on outputs of three classification models. Experimental results tested on three multiply-distorted and six singly-distorted image quality databases show that MUSIQUE-II yields not only a substantial improvement in quality predictive performance as compared with its predecessor, but also highly competitive performance relative to other state-of-the-art FR/NR IQA algorithms.

Over the past couple of decades, numerous image quality assessment (IQA) algorithms have been developed to estimate the quality of images that contain a single type of distortion. Although in practice, images can be contaminated by multiple distortions, previous research on quality assessment of multiply-distorted images is very limited. In this paper, we propose an efficient algorithm to blindly assess the quality of both multiply and singly distorted images based on predicting the distortion parameters using a bag of natural scene statistics (NSS) features. Our method, called MUltiply- and Singlydistorted Image QUality Estimator (MUSIQUE), operates via three main stages. In the first stage, a two-layer classification model is employed to identify the distortion types (i.e., Gaussian blur, JPEG compression, and white noise) that may exist in an image. In the second stage, specific regression models are employed to predict the three distortion parameters (i.e., σG for Gaussian blur, Q for JPEG compression, and σN for white noise) by learning the different NSS features for different distortion types and combinations. In the final stage, the three estimated distortion parameter values are mapped and combined into an overall quality estimate based on quality-mapping curves and the most-apparent-distortion strategy. Experimental results tested on three multiply-distorted and seven singly-distorted image quality databases demonstrate that the proposed MUSIQUE algorithm can achieve better/competitive performance as compared to other state-of-the-art FR/NR IQA algorithms.

The recent popularity of remote desktop software and live streaming of composited video has given rise to a growing number of applications which make use of so-called screen content images that contain a mixture of text, graphics, and photographic imagery. Automatic quality assessment (QA) of screen-content images is necessary to enable tasks such as quality monitoring, parameter adaptation, and other optimizations. Although QA of natural images has been heavily researched over the last several decades, QA of screen content images is a relatively new topic. In this paper, we present a QA algorithm, called convolutional neural network (CNN) based screen content image quality estimator (CNN-SQE), which operates via a fuzzy classification of screen content images into plain-text, computergraphics/ cartoons, and natural-image regions. The first two classes are considered to contain synthetic content (text/graphics), and the latter two classes are considered to contain naturalistic content (graphics/photographs), where the overlap of the classes allows the computer graphics/cartoons segments to be analyzed by both text-based and natural-image-based features. We present a CNN-based approach for the classification, an edge-structurebased quality degradation model, and a region-size-adaptive quality-fusion strategy. As we will demonstrate, the proposed CNN-SQE algorithm can achieve better/competitive performance as compared with other state-of-the-art QA algorithms.

Algorithms for no-reference (NR) stereoscopic image quality assessment (SIQA) aim to evaluate the perceptual quality of a stereoscopic/3D image without the assistance of its reference. Current NR SIQA models often require training on 3D distorted images and their associated human opinion scores, which ultimately restrict their further application. In this paper, we present a simple yet effective NR SIQA model that does not require training on existing 3D image databases. Instead, we train our model on a large dataset of natural stereoscopic images based on learning the local statistics of the Cyclopean contrast maps, and then use the existing 2D NR IQA model to help guide the NR SIQA task. Experimental results demonstrate the efficacy of our proposed method.

Algorithms for a stereoscopic image quality assessment (IQA) aim to estimate the qualities of 3D images in a manner that agrees with human judgments. The modern stereoscopic IQA algorithms often apply 2D IQA algorithms on stereoscopic views, disparity maps, and/or cyclopean images, to yield an overall quality estimate based on the properties of the human visual system. This paper presents an extension of our previous 2D most apparent distortion (MAD) algorithm to a 3D version (3D-MAD) to evaluate 3D image quality. The 3D-MAD operates via two main stages, which estimate perceived quality degradation due to 1) distortion of the monocular views and 2) distortion of the cyclopean view. In the first stage, the conventional MAD algorithm is applied on the two monocular views, and then the combined binocular quality is estimated via a weighted sum of the two estimates, where the weights are determined based on a block-based contrast measure. In the second stage, intermediate maps corresponding to the lightness distance and the pixel-based contrast are generated based on a multipathway contrast gain-control model. Then, the cyclopean view quality is estimated by measuring the statistical-difference-based features obtained from the reference stereopair and the distorted stereopair, respectively. Finally, the estimates obtained from the two stages are combined to yield an overall quality score of the stereoscopic image. Tests on various 3D image quality databases demonstrate that our algorithm significantly improves upon many other state-of-the-art 2D/3D IQA algorithms.