In this paper, we present a novel organ-branched CNN method for face super-resolution, named OBC-FSR. It is the first work focusing on facial-part-specific face SR, which consists of a local (facial part) network and a global network. Specifically, local network enhances the five key regions of human faces separately by Wasserstein generative adversarial networks (WGAN). Simultaneously, it also predicts five key regions’ masks, namely, eyes, eyebrows, mouth, nose, and other parts. The output of the local network is obtained by merging super-resolved five key regions. In order to alleviate boundary effects and distortions in the result of local network, our proposed network also includes a global network, which learns the direct mapping between LR and HR human faces. The final HR result of our FSR method is a fusion of the out-puts of local and global networks. Experimental results verify the superior performance of our method compared to the state-of-the-art.

The removal of non-uniform motion blur, which is caused by the jittering of the camera and the rapid movement of the target objects, is quite a challenging task. A majority of deblurring works focus on the traditional two-step method: firstly, the motion kernel estimation, and then the energy function minimization. In this paper, we address the problem of single image non-uniform motion blur with deep convolutional neural network (CNN) in a decent end-to-end manner. We proposed a brand-new convolution architecture named as 'hole convolution', of which the kernel takes a rectangular ring of the neighbors to the center pixel into computation and thus the reception field is greatly expanded. Moreover, we present a scale-aware convolutional neural network to recover the latent sharp image. With the carefully designed units, the hole convolution layers as well as the exquisite network structure, our proposed deep CNN can eliminate blur from various scales without upsampling and downsampling. Experimental results show that our method can effectively restore the latent sharp image, which outperforms the state-of-the-art method by a large margin.

Single-image super-resolution (SISR) is a classic problem in the image processing community, which aims at generating a high-resolution image from a low-resolution one. In recent years, deep learning based SISR methods emerged and achieved a performance leap than previous methods. However, because the evaluation metrics of SISR methods is peak signal-to-noise ratio (PSNR), previous methods usually choose L2-norm as the loss function. This leads to a significant improvement in the final PSNR value but little improvement in perceptual quality. In this paper, in order to achieve better results in both perceptual quality and PSNR values, we propose an objective quality assessment driven SISR method. First, we propose a novel full-reference image quality assessment approach for SISR and employ it as a loss function, namely super-resolution image quality assessment (SR-IQA) loss. Then, we combine SR-IQA loss with L2-norm to guide our proposed SISR method to achieve better results. Besides that, our proposed SISR method consists of several proposed highway units. Furthermore, in order to verify the generalization ability of our new kind of loss function, we integrate SR-IQA loss to generative adversarial networks based SR method and achieve better perceptual quality. Experimental results prove that our proposed SISR method achieves better performance than other methods both qualitatively and quantitatively in most of the cases.

Single-image super-resolution (SISR) is a very important and classic problem of the computer vision community. Although a lot of SISR methods have been proposed, few studies have been conducted to address the quality assessment of SISR methods. In this paper, we proposed a deep learning based no-reference image quality assessment (NR-IQA) model for SISR. We took small patches from images to form our training set and labeled them with different scores. With the aid of well-designed architecture and training strategy, our method achieved a performance leap than state-of-the-art methods. Experimental results proved the generalizability and the effectiveness of the proposed model.

Face hallucination aims to generate a high resolution face from a low resolution one. Generic super resolution methods can not solve this problem well, because human face has a strong structure. With the rapid development of the deep learning technique, some convolutional neural networks (CNNs) models for face hallucination emerged and achieved state-of-the-art performance. In this paper, we proposed a five-branch network based on five key parts of human face. Each branch of this network aims to generate a high resolution key part. The final high resolution face is the combination of the five branches' output. In addition, we designed a gated enhance unit (GEU) and cascade it to form our network architecture. Experimental results confirm that our method can generate pleasing high resolution faces.

Convolutional neural networks (CNNs) have been widely used in computer vision community. Single image super-resolution (SISR) is a classic computer vision problem, which aims to output a high-resolution image from a low-resolution one. In recent years, CNNs-based SISR methods emerged and achieved a performance leap. In this paper, we present a highly accurate deep CNNs model for SISR. Inspired by the ideas in highway networks, we propose a highway unit and cascade highway units to ensemble our model. Furthermore, we employ structural similarity index (SSIM) as a part of loss function to enhance the accuracy of trained deep CNNs model. Experimental results show that our proposed model outperforms other state-of-the-art methods.

The goal of image quality assessment (IQA) is to use computational models to measure the consistency between image quality and subjective evaluations. In recent years, convolutional neural networks (CNNs) have been widely used in image processing community and have achieved performance leaps than non CNNs-based methods. In this work, we describe an accurate deep CNNs model for no-reference IQA. Taking image patches as input, our deep CNNs model achieves an end-to-end method without any handcrafted features and pre-processing procedures that are employed by previous no-reference IQA methods. The proposed model consists of six convolutional layers, two fully connected layers, one max pooling layer and two sum layers. The experimental results verify that our model outperforms the state-of-the-art no-reference IQA methods and most of the full-reference IQA metrics.

Convolutional neural networks (CNNs) have been widely used in image processing community. Image deblocking is a post-processing strategy, which aims to reduce the visually annoying blocking artifacts that are caused by block-based transform coding at low bit rates. In recent years, CNNs based methods have been proposed to solve this classic image processing problem. In this paper, we present an efficient deep C-NNs model for image deblocking. Our model can well alleviate the conflict between bit reduction and quality preservation by taking local small patches into consideration. Our trained model can be used to deblock lossy compressed images with different quality factors. The proposed model can be easily integrated into the existing codecs as a post-processing procedure without changing the codec. Experimental results verify that our proposed model outperforms the state-of-the-art methods in both the objective quality and the perceptual quality.

Traditional image resizing methods, such as uniform scaling and content-aware image retargeting, are designed to preserve the visually salient contents of an image while resizing it. In this paper, we propose a novel image resizing approach called recognition-oriented image retargeting. Its goal is to preserve the distinctive local features for recognition instead of the traditional visual saliency during resizing. Moreover, we also apply our approach to image matching and image retrieval applications to verify its performance. Meanwhile, using our approach to these applications is able to solve some of the challenging problems in their fields. In image matching application, we find that our approach shows promising preservation of local feature descriptors. In image retrieval task, extensive experiments on Oxford5K, Holidays, Paris, and Flickr100k data sets demonstrate that our approach consistently outperforms other image retargeting methods by large margins in the aspects of retrieval precision and query bits.

With a growth of surveillance cameras, the amount of captured videos expands. Manually analyzing and retrieving surveillance video is labor intensive and expensive. It would be much more convenient to generate a video digest, with which we can view the video in a fast and motion-preserving way. In this paper, we propose a novel video synopsis approach to generate condensed video, which uses an object tracking method for extracting important objects. This method will generate video tubes and a seam carving method to condense the original video. Experimental results demonstrate that our proposed method can achieve a high condensation rate while preserving all the important objects of interest. Therefore, this approach can enable users to view the surveillance video with great efficiency.

With the sharp increasing of mobile devices, conducting search on mobile devices becomes pervasive, and one of the most popular applications is mobile visual search. To achieve low bit-rate visual search, most of the existing works focus on addressing local descriptor coding and BoW histogram compression . In this paper, we extend the concept of image retargeting and propose a new image resizing approach that is devoted to preserving the robust local features in the query image while resizing it. Based on the extended concept, we introduce a novel mobile-visual-search scheme that conducts the proposed approach to reduce the size of the query image for achieving low bit-rate visual search. Extensive experiments on Oxford 5 K and Flickr 100k datasets show that our approach obtains superior retrieval performance than state-of-the-art image resizing approaches at the similar query size; meanwhile, it is cost effective in terms of processing time.

Image retargeting attempts to adapt images to different devices while preserving the salient contents. Most existing methods address retargeting of a single image. In this paper, we propose a novel image retargeting method for resizing a pair of stereo images. Naively retargeting each image independently will distort the geometric structure and will impair the perception of the 3-D structure of the scene. We introduce an extension to the 2-D image retargeting method that works on a pair of stereo images. We demonstrate the performance of our method on a number of challenging indoor and outdoor stereo images. Experimental results show that our method is able to provide visually comfortable resized images when the resizing ratio is relatively high.

This paper presents a new image retargeting method based on seam searching and pixel fusion. First, the importance map of the original image, which is represented by the saliency of every pixel, is generated. Second, all the pixels in the original image are divided into W (width of the original image) groups with the help of seam searching. Then, it performs inter-row coherence filtering on importance map to maintain the spatial coherence, and directly utilizes the filtered importance map to generate the scaling map. Finally, after pixel fusion with the scaling map, the image can be retargeted effectively. Experimental results show that our method is able to provide visually pleasing resized images.

With the increasing requirement of efficient image retargeting, many algorithms have been proposed for adapting images contents to various display settings. However, most of these algorithms work in spatial domain of raw images. Since images are mostly stored in DCT-based compressed format such as JPEG, it will be very attractive to realize the image retargeting in compressed domain. In this paper, we propose a new low complexity DCT-based image retargeting method, which is completely performed in compressed domain. This proposed algorithm is based on the construction of block level importance map and calculation of block level forward energy. Experimental results prove that our proposed algorithm is able to significantly reduce image decoding complexity as well as image retargeting complexity, while providing the satisfying image retargeting quality compared with other methods.

Content-aware image retargeting enables images to fit different devices with various aspect ratios while preserving salient contents. Meanwhile, assessing the quality of image retargeting and unifying both subjective and objective evaluation have become a prominent challenge. In this paper, we propose an image quality assessment based on Radial Basis Function (RBF) neural network. We propose a new feature of image retargeting evaluation, which adapts structural similarity (SSIM) and saliency. By also including other existing features, we build a neural network to assess the quality of the retargeted image. The neural network is trained to combine the above-mentioned features. The accuracy of our proposed assessment is verified by simulations and it possesses huge practical significance.

This paper presents a new image retargeting method, which is able to provide both high efficiency and quality. Firstly, our method calculates the image resizing factors in both horizontal and vertical directions. Then, it resizes the image in these both directions sequentially in order to achieve the target aspect ratio with the help of our proposed mapping functions. Finally, the reconstructed image can be uniformly scaled to the target resolution. Experimental results show that our method is not only cost effective in terms of computational resource, but also provides visually pleasing resized images.