In this paper, we present a novel organ-branched CNN method for face super-resolution, named OBC-FSR. It is the first work focusing on facial-part-specific face SR, which consists of a local (facial part) network and a global network. Specifically, local network enhances the five key regions of human faces separately by Wasserstein generative adversarial networks (WGAN). Simultaneously, it also predicts five key regions’ masks, namely, eyes, eyebrows, mouth, nose, and other parts. The output of the local network is obtained by merging super-resolved five key regions. In order to alleviate boundary effects and distortions in the result of local network, our proposed network also includes a global network, which learns the direct mapping between LR and HR human faces. The final HR result of our FSR method is a fusion of the out-puts of local and global networks. Experimental results verify the superior performance of our method compared to the state-of-the-art.

Under stereo settings, the problems of disparity estimation, stereo magnification and stereo-view synthesis have gathered wide attention. However, the limited image quality brings non-negligible difficulties in developing related applications and becomes the main bottleneck of stereo images. To the best of our knowledge, stereo image restoration is rarely studied. Towards this end, this paper analyses how to effectively explore disparity information, and proposes a unified stereo image restoration framework. The proposed framework explicitly learn the inherent pixel correspondence between stereo views and restores stereo image with the cross-view information at image and feature level. A Feature Modulation Dense Block (FMDB) is introduced to insert disparity prior throughout the whole network. The experiments in terms of efficiency, objective and perceptual quality, and the accuracy of depth estimation demonstrates the superiority of the proposed framework on various stereo image restoration tasks.

Image retargeting aims to obtain high visual quality of target images for human vision. Through semantic segmentation and understanding of input images, we can better preserve the important semantic regions, so as to effectively improve the performance of image retargeting. Benefit from the successful application of deep neural network in the field of semantic segmentation, in this paper, we propose a novel image retargeting approach using semantic segmentation and pixel fusion. Compared with existing image retargeting methods, our approach can effectively reduce geometric distortion during image retargeting by finely reallocating scaling factors for each region based on the semantic segmentation results. Experimental results demonstrate that the proposed approach can well preserve important semantic regions while leaving less unnatural geometric distortion. Our approach also shows the important role of semantic segmentation and understanding of scenes in image retargeting in detail.

Recently, significant progress has been witnessed in image restoration benefited from the development of deep convolutional neural networks (CNN). However, we note that many state-of-the-art image restoration networks can be unfolded as a one-level architecture, which is constructed by stacking multiple convolution layers or blocks. As the depth of network grows, the information flow is weakened. And, existing skip connection has restricted ability to pass the previous state to the latter layers in the network. Based on above observations, we explore a multi-level aggregated network (MLAN) to fully exploit features of deeper layers. The proposed MLAN can extract more features by merging layers at different levels progressively, and can better aggregate the features by the augmented connection manner. Experimental results demonstrate a satisfactory performance of the proposed model on different image restoration tasks.

Single-image super-resolution (SISR) is a classic problem in the image processing community, which aims at generating a high-resolution image from a low-resolution one. In recent years, deep learning based SISR methods emerged and achieved a performance leap than previous methods. However, because the evaluation metrics of SISR methods is peak signal-to-noise ratio (PSNR), previous methods usually choose L2-norm as the loss function. This leads to a significant improvement in the final PSNR value but little improvement in perceptual quality. In this paper, in order to achieve better results in both perceptual quality and PSNR values, we propose an objective quality assessment driven SISR method. First, we propose a novel full-reference image quality assessment approach for SISR and employ it as a loss function, namely super-resolution image quality assessment (SR-IQA) loss. Then, we combine SR-IQA loss with L2-norm to guide our proposed SISR method to achieve better results. Besides that, our proposed SISR method consists of several proposed highway units. Furthermore, in order to verify the generalization ability of our new kind of loss function, we integrate SR-IQA loss to generative adversarial networks based SR method and achieve better perceptual quality. Experimental results prove that our proposed SISR method achieves better performance than other methods both qualitatively and quantitatively in most of the cases.

No-reference image quality assessment (NR-IQA) is a non-trivial task, because it is hard to find a pristine counterpart for an image in real applications, such as image selection, high quality image recommendation, etc. In recent years, deep learning-based NR-IQA methods emerged and achieved better performance than previous methods. In this paper, we present a novel deep neural networks-based multi-task learning approach for NR-IQA. Our proposed network is designed by a multi-task learning manner that consists of two tasks, namely, natural scene statistics (NSS) features prediction task and the quality score prediction task. NSS features prediction is an auxiliary task, which helps the quality score prediction task to learn better mapping between the input image and its quality score. The main contribution of this work is to integrate the NSS features prediction task to the deep learning-based image quality prediction task to improve the representation ability and generalization ability. To the best of our knowledge, it is the first attempt. We conduct the same database validation and cross database validation experiments on LIVE1, TID20132, CSIQ3, LIVE multiply distorted image quality database (LIVE MD)4, CID20135, and LIVE in the wild image quality challenge (LIVE challenge)6databases to verify the superiority and generalization ability of the proposed method. Experimental results confirm the superior performance of our method on the same database validation; our method especially achieves 0.984 and 0.986 on the LIVE image quality assessment database in terms of the Pearson linear correlation coefficient (PLCC) and Spearman rank-order correlation coefficient (SROCC), respectively. Also, experimental results from cross database validation verify the strong generalization ability of our method. Specifically, our method gains significant improvement up to 21.8% on unseen distortion types.

Identifying small size images or small objects is a notoriously challenging problem, as discriminative representations are difficult to learn from the limited information contained in them with poor-quality appearance and unclear object structure. Existing research works usually increase the resolution of low-resolution image in the pixel space in order to provide better visual quality for human viewing. However, the improved performance of such methods is usually limited or even trivial in the case of very small image size (we will show it in this paper explicitly). In this paper, different from image super-resolution (ISR), we propose a novel super-resolution technique called feature super-resolution (FSR), which aims at enhancing the discriminatory power of small size image in order to provide high recognition precision for machine. To achieve this goal, we propose a new Feature Super-Resolution Generative Adversarial Network (FSR-GAN) model that transforms the raw poor features of small size images to highly discriminative ones by performing super-resolution in the feature space. Our FSR-GAN consists of two subnetworks: a feature generator network G and a feature discriminator network D. By training the G and the D networks in an alternative manner, we encourage the G network to discover the latent distribution correlations between small size and large size images and then use G to improve the representations of small images. Extensive experiment results on Oxford5K, Paris, Holidays, and Flick100k datasets demonstrate that the proposed FSR approach can effectively enhance the discriminatory ability of features. Even when the resolution of query images is reduced greatly, e.g., 1/64 original size, the query feature enhanced by our FSR approach achieves surprisingly high retrieval performance at different image resolutions and increases the retrieval precision by 25% compared to the raw query feature.

Single-image super-resolution (SISR) is a very important and classic problem of the computer vision community. Although a lot of SISR methods have been proposed, few studies have been conducted to address the quality assessment of SISR methods. In this paper, we proposed a deep learning based no-reference image quality assessment (NR-IQA) model for SISR. We took small patches from images to form our training set and labeled them with different scores. With the aid of well-designed architecture and training strategy, our method achieved a performance leap than state-of-the-art methods. Experimental results proved the generalizability and the effectiveness of the proposed model.

Face hallucination aims to generate a high resolution face from a low resolution one. Generic super resolution methods can not solve this problem well, because human face has a strong structure. With the rapid development of the deep learning technique, some convolutional neural networks (CNNs) models for face hallucination emerged and achieved state-of-the-art performance. In this paper, we proposed a five-branch network based on five key parts of human face. Each branch of this network aims to generate a high resolution key part. The final high resolution face is the combination of the five branches' output. In addition, we designed a gated enhance unit (GEU) and cascade it to form our network architecture. Experimental results confirm that our method can generate pleasing high resolution faces.

Convolutional neural networks (CNNs) have been widely used in computer vision community. Single image super-resolution (SISR) is a classic computer vision problem, which aims to output a high-resolution image from a low-resolution one. In recent years, CNNs-based SISR methods emerged and achieved a performance leap. In this paper, we present a highly accurate deep CNNs model for SISR. Inspired by the ideas in highway networks, we propose a highway unit and cascade highway units to ensemble our model. Furthermore, we employ structural similarity index (SSIM) as a part of loss function to enhance the accuracy of trained deep CNNs model. Experimental results show that our proposed model outperforms other state-of-the-art methods.

The goal of image quality assessment (IQA) is to use computational models to measure the consistency between image quality and subjective evaluations. In recent years, convolutional neural networks (CNNs) have been widely used in image processing community and have achieved performance leaps than non CNNs-based methods. In this work, we describe an accurate deep CNNs model for no-reference IQA. Taking image patches as input, our deep CNNs model achieves an end-to-end method without any handcrafted features and pre-processing procedures that are employed by previous no-reference IQA methods. The proposed model consists of six convolutional layers, two fully connected layers, one max pooling layer and two sum layers. The experimental results verify that our model outperforms the state-of-the-art no-reference IQA methods and most of the full-reference IQA metrics.

Convolutional neural networks (CNNs) have been widely used in image processing community. Image deblocking is a post-processing strategy, which aims to reduce the visually annoying blocking artifacts that are caused by block-based transform coding at low bit rates. In recent years, CNNs based methods have been proposed to solve this classic image processing problem. In this paper, we present an efficient deep C-NNs model for image deblocking. Our model can well alleviate the conflict between bit reduction and quality preservation by taking local small patches into consideration. Our trained model can be used to deblock lossy compressed images with different quality factors. The proposed model can be easily integrated into the existing codecs as a post-processing procedure without changing the codec. Experimental results verify that our proposed model outperforms the state-of-the-art methods in both the objective quality and the perceptual quality.

Image retargeting attempts to adapt images to different devices while preserving the salient contents. Most existing methods address retargeting of a single image. In this paper, we propose a novel image retargeting method for resizing a pair of stereo images. Naively retargeting each image independently will distort the geometric structure and will impair the perception of the 3-D structure of the scene. We introduce an extension to the 2-D image retargeting method that works on a pair of stereo images. We demonstrate the performance of our method on a number of challenging indoor and outdoor stereo images. Experimental results show that our method is able to provide visually comfortable resized images when the resizing ratio is relatively high.

Content-aware image retargeting enables images to fit different devices with various aspect ratios while preserving salient contents. Meanwhile, assessing the quality of image retargeting and unifying both subjective and objective evaluation have become a prominent challenge. In this paper, we propose an image quality assessment based on Radial Basis Function (RBF) neural network. We propose a new feature of image retargeting evaluation, which adapts structural similarity (SSIM) and saliency. By also including other existing features, we build a neural network to assess the quality of the retargeted image. The neural network is trained to combine the above-mentioned features. The accuracy of our proposed assessment is verified by simulations and it possesses huge practical significance.