The performance of deep learning based image super-resolution (SR) methods depend on how accurately the paired low and high resolution images for training characterize the sampling process of real cameras. Low and high resolution (LR ∼ HR) image pairs synthesized by degradation models (e.g., bicubic downsampling) deviate from those in reality; thus the synthetically-trained DCNN SR models work disappointingly when being applied to real-world images. To address this issue, we propose a novel data acquisition process to shoot a large set of LR ∼ HR image pairs using real cameras. The images are displayed on an ultra-high quality screen and captured at different resolutions. The resulting LR ∼ HR image pairs can be aligned at very high sub-pixel precision by a novel spatial-frequency dual-domain registration method, and hence they provide more appropriate training data for the learning task of super-resolution. Moreover, the captured HR image and the original digital image offer dual references to strengthen supervised learning. Experimental results show that training a super-resolution DCNN by our LR ∼ HR dataset achieves higher image quality than training it by other datasets in the literature. Moreover, the proposed screen-capturing data collection process can be automated; it can be carried out for any target camera with ease and low cost, offering a practical way of tailoring the training of a DCNN SR model separately to each of the given cameras.

Subitizing, or the sense of small natural numbers, is an innate cognitive function of humans and primates; it responds to visual stimuli prior to the development of any symbolic skills, language or arithmetic. Given successes of deep learning (DL) in tasks of visual intelligence and given the primitivity of number sense, a tantalizing question is whether DL can comprehend numbers and perform subitizing. But somewhat disappointingly, extensive experiments of the type of cognitive psychology demonstrate that the examples-driven black box DL cannot see through superficial variations in visual representations and distill the abstract notion of natural number, a task that children perform with high accuracy and confidence. The failure is apparently due to the learning method not the CNN computational machinery itself. A recurrent neural network capable of subitizing does exist, which we construct by encoding a mechanism of mathematical morphology into the CNN convolutional kernels. Also, we investigate, using subitizing as a test bed, the ways to aid the black box DL by cognitive priors derived from human insight. Our findings are mixed and interesting, pointing to both cognitive deficit of pure DL, and some measured successes of boosting DL by predetermined cognitive implements. This case study of DL in cognitive computing is meaningful for visual numerosity represents a minimum level of human intelligence.

A great number of invaluable historical photographs unfortunately only exist in the form of halftone prints in old publications such as newspapers or books. Their original continuous-tone films have long been lost or irreparably damaged. There have been attempts to digitally restore these vintage halftone prints to the original film quality or higher. However, even using powerful deep convolutional neural networks, it is still difficult to obtain satisfactory results. The main challenge is that the degradation process is complex and compounded while little to no real data is available for properly training a data-driven method. In this research, we adopt a novel strategy of two-stage deep learning, in which the restoration task is divided into two stages: the removal of printing artifacts and the inverse of halftoning. The advantage of our technique is that only the simple first stage requires unsupervised training in order to make the combined network generalize on real halftone prints, while the more complex second stage of inverse halftoning can be easily trained with synthetic data. Extensive experimental results demonstrate the efficacy of the proposed technique for real halftone prints; the new technique significantly outperforms the existing ones in visual quality.

Despite extensive research efforts, blind image deblurring remains a challenge without general robust solutions. A long-overlooked problem of existing deblurring methods is that they are all designed to work on fully sampled RGB input images for simplicity. But, in practice, most RGB color images are reconstructed from Bayer mosaic data hence riddled with various high-frequency demosaicking artifacts, such as zippering and moiré patterns, which can easily derail a deblurring algorithm. In this paper, we propose a novel multi-scale deep convolutional neural network to solve demosaicking and deblurring jointly. By processing Bayer raw images directly, our method is free of the interference of demosaicking artifacts. Extensive experiments show that the joint approach greatly outperforms the simple cascade of state-of-art demosaicking and deblurring methods.

This paper presents a generic screening algorithm for expediting conventional template matching techniques. The algorithm can rule out regions with no possible matches with minimum computational efforts; moreover, the match is robust against rotation changes. The computational efficiency and robustness against rotation are gained by using a novel octagonal star shaped query template and the inclusion-exclusion principle to extract and compare patch features. Extensive experiments demonstrate that the proposed algorithm greatly reduces the search space without adversely affecting the matching accuracy.

High dynamic range (HDR) tone mapping is formulated as an optimization problem of maximizing perceivable spatial details given the limited dynamic range of display devices. This objective can be attained, as supported by our results, by a novel image display methodology called locally adaptive rank-constrained optimal tone mapping (LARCOTM). The scientific basis for LARCOTM is that the maximum discrimination power of human vision system can only be achieved in a relatively small locality of an image. LARCOTM is fundamentally different from existing HDR tone mapping techniques in that the former can preserve pixel value order statistics within localities in which human foveal vision retains maximum sensitivity, while the latter cannot. As a result, images enhanced by LARCOTM are free of artifacts such as halos and double edges that plague other HDR methods.

Image resolution in modern video processing and display systems are rocketing up in recent years. With the main stream video quality evolving from standard definition to high-definition, and further towards the emerging super high-definition, the bandwidth and power consumption of external memory are becoming serious bottlenecks. In this paper, frequency domain analysis is made on image down-sampling, which gives birth to the optimal sampling strategy for high-frequency energy protection. On that basis, we develop a new embedded compression (EC) technique, which encodes image frames through content-adaptive down-sampling and decodes them using side-information aided up-sampling. Apart from the low encoding complexity and even lower decoding complexity, the proposed EC algorithm also features high fidelity for images with sharp edges. Comparisons with some existing EC algorithms show the advantage of the new technique over its counterparts on a wide class of testing images.

Guided image filter (GIF) is an edge-preserving filtering technique that smooths the fine texture of an input image with the guide of a second image. One shortcoming of GIF and all its existing variants, such as, weighed guided image filter and gradient domain guided image filter, is that they only use one grayscale image as the guide and are consequently unable to fully utilize the rich information offered by multichannel images. In this paper, we extend GIF for multichannel guidance image and propose a novel correlation detection technique for retaining sharp edges with opposite gradient directions in the different channels of the same guidance image. Experimental results show that the proposed method preserves the details from both the input and guidance images better than existing GIF techniques.

Quantization in discrete cosine transform (DCT) domain is a widely used lossy compression technique in international multimedia compression standards from audio (e.g., MP3) to image (e.g., JPEG) to video (e.g., H.264). Unlike many degradation sources, like sensor noises, quantization errors of DCT coefficients are signal dependent and difficult to isolate and remove, causing serious artifacts in decompressed and post-processed signals. In this research, quantization errors in the DCT domain are analytically assessed. Our analysis exposes complex behaviors of the DCT quantization errors after being mapped back into the temporal or spatial domain. These behaviors are highly sensitive to quantization precision, the amplitude and phase of the input signal. Based on these observations, we develop a DCT-domain error model to predict and quantify the quantization effects in cases where artifacts are most perceivable to humans, and offer some insights into possible strategies for further suppressing compression noises.

This work is concerned with the protection of intellectual property rights and privacy against unpermitted uses of digital cameras. A technique of multispectral coded illumination (MSCI) with LED lights is proposed to defeat cameras capturing indoor scenes by inducing annoying color artifacts into the acquired images or video frames. The main idea of MSCI is to temporally modulate LED lights of different colors at certain frequencies so that they interfere with the rolling shutter of the camera, but at the same time the coded illumination appears to human eyes the same as steady white lighting.

Pairing displays and cameras can open up convenient and "free" visible light communication channels. But in realistic settings, the synchronization between displays (transmitters) and cameras (receivers) can be far more involved than assumed in the literature. This study aims to analyze and model the temporal behaviors of displays and cameras to make the visible light communication channel between the two more robust, while maintaining perceptual transparency of the transmitted data.

This paper opens up an unforeseen and intriguing application area of our recent pioneer work of temporal psychovisual modulation: wireless optical communication. It is demonstrated how a high-speed optoelectronic display functions as a 2D array of optical transmitters and at the same exhibits conventional images as usual. At the receiving end, digital cameras can download data via the optical MIMO link while user(s) can work and read the display as they are accustomed. The said unification of information display and wireless optical communication is made possible by psychovisually based image processing.

Recently, Wu and Shu proposed an intriguing technique that transmits data from regular information display devices to smart phone cameras without using any additional hardware or interfering the original content intended to display. This preliminary work reveals the possibility of conducting high-speed data transmission with displays in transparency to human eyes; however, in practice, it could introduce visible artifacts and the display-camera synchronization remains an issue that affects data throughput. In this paper, we examine this technique in more realistic application scenarios and put forward a practical solution that hides the data more effectively and greatly improves the robustness of the transmission. Experimental results show that, using the proposed solution, viewers can hardly notice any artifacts caused by the data embedded in videos, and the bit error rate is acceptable under common use conditions.

This paper proposes a new image enhancement algorithm in a recently published framework of optimal contrast-tone mapping (OCTM). The new algorithm represents a fundamental departure from traditional histogram-based image enhancement techniques (i.e., histogram equalization and all of its variants), in that second-order rather than first-order statistics is used. Perceptual quality attributes, such as contrast and tone, are quantified by joint distribution of the values of spatially adjacent pixels instead of histograms as of today. The problem of image enhancement is then formulated as one of linear programming, at the heart of which is a joint distribution-based objective function that can accommodate various psychovisual properties related to image quality. The new linear program algorithm for image enhancement is implemented and its superior performance in visual quality is empirically verified, corroborating with our analysis.

This work presents a method to determine the optimal backlight scanning signals to minimize crosstalk for time-sequential stereoscopic 3D on LCD TV with active shutter glasses. The solution is obtained through optimization of the variables defined by a model of backlight scanning that considers important aspects like liquid crystal transitions and light diffusion, subject to constraints that ensure the rendition of a uniform backlight. Compared with basic backlight scanning, the proposed method can increase luminance at a given crosstalk level or reduce crosstalk at a given luminance level.

It was recently shown that a high-speed optoelectronic display, via a novel signal processing technique called temporal psychovisual modulation (TPVM), can exhibit stereoscopic images to viewers wearing 3-D glasses and clean 2-D images to those without glasses all at the same time. This research aims to improve the above backward-compatible stereoscopy method by adjusting the backlight of today's common LED-lit liquid crystal display systems. Visual quality of the system can be enhanced by jointly optimizing the backlight intensity and the image signal at a negligible extra computational cost. For real-time applications of low-cost consumer electronics, this work also provides a low-complexity solution of backward-compatible stereoscopic display, in which highest 3-D quality is ensured with a small compromise of the 2-D quality.

Light emitting diode (LED)-backlit liquid crystal displays (LCDs) hold the promise of improving image quality while reducing the energy consumption with signal-dependent local dimming. However, most existing local dimming algorithms are mostly motivated by simple implementation, and they often lack concern for visual quality. To fully realize the potential of LED-backlit LCDs and reduce the artifacts that often occur in current systems, we propose a novel local dimming technique that can achieve the theoretical highest fidelity of intensity reproduction in either l(1) or l(2) metrics. Both the exact and fast approximate versions of the optimal local dimming algorithm are proposed. Simulation results demonstrate superior performances of the proposed algorithm in terms of visual quality and power consumption.