Current low-light light-field (LF) image enhancement algorithms tend to produce blurry results, for (1) loss of spatial details during enhancement and (2) inefficient exploitation of angular correlations, which helps to recover spatial details. Therefore, in this article, we propose a parallel multi-scale network (PMSNet), which attempts to (1) process features of different scales in parallel to aggregate the different contributions of multi-scale features at each layer, thus fully preserve spatial details, and (2) integrate multi-resolution 3D convolution streams to efficiently utilize angular correlations. Specifically, PMSNet consists of three stages: Stage-I employs multi-scale modules (MSMs) to generate local understanding with the aid of adjacent views. Notably, MSM retains high-resolution feature extraction to minimize loss of spatial details. Stage-II processes all views to encode global information. Based on the above extracted local and global information, Stage-III utilizes 3D multi-scale modules (3D-MSMs) to efficiently exploit angular correlations. To validate our idea, we comprehensively evaluate the performance of PMSNet on three publicly available datasets. Experimental results show that our method is superior to the current state-of-the-art methods, achieving an average PSNR of 24.76 dB.

Existing CNN-based light field de-occlusion (LF-DeOcc) methods suffer from occlusion removal performance degradation in the presence of large-size occlusions. In this paper, we infer that it is possibly caused by the limited receptive field of CNN and experimentally demonstrate that the de-occlusion performance is high-related to the receptive field. Therefore, a novel LF-DeOcc network based on Swin Transformer and CNN, which aims to exploit both global and local receptive fields, is firstly proposed for light field de-occlusion task. CNNs are employed at shallow layers to compensate for the deficiency of Transformers in extracting local features, while Transformers are employed at deep layers to capture the global patterns of large size occlusions. Hence, by integrating the global and local features, one could restore occlusion-free images effectively. For performance evaluation, a large dataset with mild-to-severe occlusions is developed and tested, with average occlusion rates of 19.82%, 32.13% and 40.56%, respectively. Experimental results show that the proposed network is superior to state-of-the-art methods, achieving 27.87 dB and 29.46 dB on the public dataset and our developed dataset, respectively. Finally, a new evaluation method has been presented in our work, i.e., by utilizing the real target detection task to evaluate the performance of LF de-occlusion algorithms. The practicability of our algorithm is validated using the new evaluation method.

Video captioning refers to automatic generate natural language sentences, which summarize the video contents. Inspired by the visual attention mechanism of human beings, temporal attention mechanism has been widely used in video description to selectively focus on important frames. However, most existing methods based on temporal attention mechanism suffer from the problems of recognition error and detail missing, because temporal attention mechanism cannot further catch significant regions in frames. In order to address above problems, we propose the use of a novel spatial-temporal attention mechanism (STAT) within an encoder-decoder neural network for video captioning. The proposed STAT successfully takes into account both the spatial and temporal structures in a video, so it makes the decoder to automatically select the significant regions in the most relevant temporal segments for word prediction. We evaluate our STAT on two well-known benchmarks: MSVD and MSR-VTT-10K. Experimental results show that our proposed STAT achieves the state-of-the-art performance with several popular evaluation metrics: BLEU-4, METEOR, and CIDEr.

Presents corrections to affiliations in the above named paper.

This paper presents a method for stereoscopic image stitching, which can make stereoscopic images look as natural as possible. Our method combines a constrained projective warp and a shape-preserving warp to reduce the projective distortion and the vertical disparity of the stitched image. In addition to provide a good alignment accuracy and maintain the consistency of input stereoscopic images, we add a specific restriction into the projective warp, which establishes the connection between target left and right images. To optimize the whole warp, a energy term is designed. It can constrain the shape of straight line and vertical disparity. Experimental results on a variety of stereoscopic images can ensure the efficiency of the proposed method.

We propose a novel method for multi-person 2D pose estimation. Our model zooms in the image gradually, which we refer to as the Magnify-Net, to solve the bottleneck problem of mean average precision (mAP) versus pixel error. Moreover, we squeeze the network efficiently by an inspired design that increases the mAP while saving the processing time. It is a simple, yet robust, bottom-up approach consisting of one stage. The architecture is designed to detect the part position and their association jointly via two branches of the same sequential prediction process, resulting in a remarkable performance and efficiency rise. Our method outcompetes the previous state-of-the-art results on the challenging COCO key-points task and MPII Multi-Person Dataset.

Patch based image modelings have shown great potential in image denoising. They mainly exploit the nonlocal self-similarity (NSS) of either input degraded images or clean natural ones when training models, while failing to learn the mappings between them. More seriously, these algorithms have very high time complexity and poor robustness when handling images with different noise variances and resolutions. To address these problems, in this paper, we propose very deeply cascaded residual networks (VDCRN) to build the precise relationships between the noisy images and their corresponding noise-free ones. It adopts a new residual unit with an identity skip connection (shortcut) to make training easy and improve generalization. The introduction of shortcut is helpful to avoid the problem of gradient vanishing and preserve more image details. By cascading three such residual units, we build the VDCRN to deploy deeper and larger convolutional networks. Based on such a residual network, our VDCRN achieves very fast speed and good robustness. Experimental results demonstrate that our model outperforms a lot of state-of-the-art denoising algorithms quantitively and qualitively.

In this paper, we propose a saliency prediction algorithm utilizing generative adversarial networks. The proposed system contains two parts: saliency network and adversarial networks. The saliency network is the basis for saliency prediction, which calculates an Euclidean cost function on the grayscale values between the predicted saliency map and the ground truth. In order to improve the accuracy of the algorithm, adversarial networks are subsequently utilized to extract the features of input data by coordinating the learning rates of the two sub-networks contained in the networks. Experimental results validate the high accuracy of the proposed approach compared with the state-of-the-art models on three public datasets, SALICON, MIT1003 and Cerf.

In this paper, we propose a novel method for 4D light-field (LF) depth estimation exploiting the special linear structure of an epipolar plane image (EPI) and locally linear embedding (LLE). Without high computational complexity, depth maps are locally estimated by locating the optimal slope of each line segmentation on the EPIs, which are projected by the corresponding scene points. For each pixel to be processed, we build and then minimize the matching cost that aggregates the intensity pixel value, gradient pixel value, spatial consistency, as well as reliability measure to select the optimal slope from a predefined set of directions. Next, a subangle estimation method is proposed to further refine the obtained optimal slope of each pixel. Furthermore, based on a local reliability measure, all the pixels are classified into reliable and unreliable pixels. For the unreliable pixels, LLE is employed to propagate the missing pixels by the reliable pixels based on the assumption of manifold preserving property maintained by natural images. We demonstrate the effectiveness of our approach on a number of synthetic LF examples and real-world LF data sets, and show that our experimental results can achieve higher performance than the typical and recent state-of-the-art LF stereo matching methods.

Block-wise intra/inter prediction, transformation and quantization used in block-based hybrid video coding will inevitably result in blocking artifacts, especially at the low bit rate. To address this problem, this paper employs a deep convolutional neural network (CNN) to approximate the reverse function of video compression, motived by the great success of deep learning in computer vision fields recently. The proposed method establishes an end-to-end mapping, represented as the CNN, which takes the decompressed frame as input and outputs the enhanced one. Employing numerous sequences compressed by H.264 and HEVC reference software, the proposed CNN learns the connections between the lossy frame and the original one in an implicit way under different quantization parameters (QP). Figure 1 shows the architecture of our CNN and the pipeline of the network training. We build our network with convolution layers and ReLU layer and the weights and biases of all the convolution layers in our model are updated by minimizing the loss using stochastic gradient descent with the standard backpropagation. We implement the CNN as a post-loop deblocking filter and explore varying CNN parameters for different QPs. Various experimental results demonstrate that the proposed method is able to significantly improve the quality of enhanced frames in terms of both objective and subjective criterions.

In this paper, we present an accurate saliency detection algorithm based on depth feature for 3D images. We first calculate depth cue based on the sharp regions' positions within the depth ranges. Then, the coarse saliency map is computed based on the background and location prior. Finally, we employ the contrast information in the coarse saliency map to obtain the final result. Experimental evaluation by comparison with existed methods verifies the effectiveness of our proposed algorithm in terms of precision, recall and F-Measure.

There is an increasing need for high-quality multimedia applications based on block-based hybrid video coding. Inevitably, the frame will degrade during the process of block-wise intra/inter prediction, transformation, and quantization, especially when the bit rate is low. In this paper, we propose an efficient decompressed video enhancement algorithm based on the adjusted anchored neighborhood regression (A+) method. In our work, first, we learn offline linear regressors, i.e. projection matrices from the decompressed to original video frames in the training phase. For grouping anchored neighborhoods more accurately, we adopt MI-KSVD rather than KSVD to learn the dictionary. Moreover, we exploit the mutual coherence between dictionary atoms and training samples to find the nearest neighbors. Second, in the enhancement phase, we boost the quality of input decompressed videos offline by learned regression priors. To verify the robustness of our enhancement method, extensive experiments are conducted. As shown in our experimental results, the proposed enhancement method yields superior performance both objectively and subjectively.

Quantitative evaluation of image segmentation quality is usually based on comparing a segmentation with multiple reference segmentations. Instead of holistically comparing with each reference, we propose a region based evaluation framework, where an exemplar reference is adaptively constructed and applied to a generally defined evaluation measure. As examples, we implement three well-known evaluation measures and present an efficient scheme to compute each measure. Extensive experiments on the benchmark databases show that the proposed evaluation framework can improve the evaluation precision of existing measures.

Local binary pattern (LBP) has successfully been used in computer vision and pattern recognition applications, such as texture recognition. It could effectively address grayscale and rotation variation. However, it failed to get desirable performance for texture classification with scale transformation. In this paper, a new method based on dominant LBP in scale space is proposed to address scale variation for texture classification. First, a scale space of a texture image is derived by a Gaussian filter. Then, a histogram of pre-learned dominant LBPs is built for each image in the scale space. Finally, for each pattern, the maximal frequency among different scales is considered as the scale invariant feature. Extensive experiments on five public texture databases (University of Illinois at Urbana-Champaign, Columbia Utrecht Database, Kungliga Tekniska Högskolan-Textures under varying Illumination, Pose and Scale, University of Maryland, and Amsterdam Library of Textures) validate the efficiency of the proposed feature extraction scheme. Coupled with the nearest subspace classifier, the proposed method could yield competitive results, which are 99.36%, 99.51%, 99.39%, 99.46%, and 99.71% for UIUC, CUReT, KTH-TIPS, UMD, and ALOT, respectively. Meanwhile, the proposed method inherits simple and efficient merits of LBP, for example, it could extract scale-robust feature for a 200×200 image within 0.24 s, which is applicable for many real-time applications.

We propose a novel light field super-resolution framework based on hybrid imaging system, which combines two different imaging mechanisms: conventional imaging and current art-of-the-state imaging - light field imaging. We take advantage of conventional imaging in spatial resolution to make up light field and reconstruct a higher quality light field. In our method, we classify the points of the 3D scene: First, for highlight and occlusion, dictionary learning based interpolation is utilized, Second, for other areas, an improved patch matching algorithm is applied. As shown in experimental results, compared with four methods, which include the art-of-the-state algorithms, our approach is effective.

Specular reflection removal is indispensable to many computer vision tasks. However, most existing methods fail or degrade in complex real scenarios for their individual drawbacks. Benefiting from the light field imaging technology, this paper proposes a novel and accurate approach to remove specularity and improve image quality. We first capture images with specularity by the light field camera (Lytro ILLUM). After accurately estimating the image depth, a simple and concise threshold strategy is adopted to cluster the specular pixels into "unsaturated" and "saturated" category. Finally, a color variance analysis of multiple views and a local color refinement are individually conducted on these two categories to recover diffuse color information. Experimental evaluation by comparison with existed methods verifies the effectiveness of our proposed algorithm.

We propose a fast single image super-resolution algorithm based on adaptive local nonparametric regression. Making use of dictionary learning and regression, we learn multiple projection matrices mapping low-resolution features to their corresponding high-resolution ones directly. Different from previous linear regression that needs some constant parameters, our method would not use extra parameters for regression. We use the mutual coherence between dictionary atom and low-resolution feature as a label to reconstruct more sophisticated high-resolution feature. As we use the same form of mutual coherence as labels in both training and testing phases, our method would lead to an adaptive local linear regression model. Moreover, we investigate the statistical property of the dictionary atoms from the training features. Utilizing the learned statistical priors, our method would not only obtain more useful dictionary atoms, but also further decrease the computational time. As shown in our experimental results, the proposed method yields high-quality super-resolution images quantitatively and visually against state-of-the-art methods.

Depth estimation from defocus (DFD) has proved to be an efficient way to recover depth information based on the blur amount of defocus images. By introducing a multi-scale strategy into DFD, a novel depth estimation method from a single defocused image is proposed in this paper. The original input image is re-blurred using Gaussian kernels with different scale parameters, then a robust estimation of defocus blur amount at edge locations could be obtained by calculating the gradient magnitude ratio according to the original and re-blurred images. Dense defocus maps are generated via global interpolation and refinement and hence depth can be obtained under certain camera parameters. Experimental results demonstrate the effectiveness of the proposed method on obtaining high quality dense defocus and depth maps.

Depth map, as important auxiliary information in 3D procession, is used to synthesize virtual view rather than exhibition. Inspired by this, a synthesis-guided depth super resolution (SGDSR) algorithm is proposed. Employing the synthesis error between virtual view and corresponding original one as the criteria, the best super-resolved result is selected among numerous candidate super resolution (SR) results. To fully exploit varying property within different regions of an image, a patch-based SGDSR is further devised in this paper. Experimental results demonstrate the effectiveness of our method subjectively and objectively on both single view and two views platform based on depth-image-based rendering (DIBR).

This paper address the problem of efficient, realtime estimation of the particulate mass concentration, exactly PM2.5 (particles with aerodynamic diameters less than 2.5 μm) from a superb view image. And the proposed method is to achieve high degree of accuracy at the cost of only modest user's effort by analyzing the relationship between the PM2.5 and the degradation of the observed image. With the fitting algorithm with experimental data, the PM2.5 could be real-time estimated by a general camera with little artificial participation, and the correlation coefficient produced by our data set and the standard observation will be as high as 0.8219, as the MSE (Mean Squared Error) value 51.2324 μg/m3.

In this paper, an in-depth analysis on the statistical distribution characteristics of human tongue color that aims to propose a mathematically described tongue color space for diagnostic feature extraction is presented. Three characteristics of tongue color space, i.e., tongue color gamut that defines the range of colors, color centers of 12 tongue color categories, and color distribution of typical image features in the tongue color gamut, are elaborately investigated in this paper. Based on a large database, which contains over 9000 tongue images collected by a specially designed noncontact colorimetric imaging system using a digital camera, the tongue color gamut is established in the CIE chromaticity diagram by an innovatively proposed color gamut boundary descriptor using one-class SVM algorithm. Thereafter, centers of 12 tongue color categories are defined accordingly. Furthermore, color distributions of several typical tongue features, such as red points and petechial points, are obtained to build a relationship between the tongue color space and color distributions of various tongue features. With the obtained tongue color space, a new color feature extraction method is proposed for diagnostic classification purposes, with experimental results validating its effectiveness.

In order to improve the correction accuracy on tongue colors by use of Munsell colorchecker, this research aims to design a new colorchecker by aid of tongue color space. Three essential issues leading to the development of this space-based colorchecker are elaborately investigated in this study. Firstly, based on a large and comprehensive tongue database, tongue color space is established by which all visible colors can be classified as tongue or non-tongue colors. Hence, colors of the designed tongue colorchecker are selected from tongue colors to achieve high correction performance. Secondly, the minimum sufficient number of colors involved in colorchecker is yielded by comparing the correction accuracy when different number (ranged from 10 to 200) of colors are contained. Thereby, 24 colors are included because the obtained minimum number of colors is 20. Lastly, criteria for optimal color selection and its corresponding objective function are presented. Two color selection methods, i.e., greedy and clustering-based selection method, are proposed to solve the objective function. Experimental results show that clustering-based one outperforms its counterpart to generate the new tongue colorchecker. Compared to Munsell colorchecker, this proposed space-based colorchecker can greatly improve the correction accuracy by 48%. Further experimental results on more correction task also validate its effectiveness and superiority.

The color images produced by digital cameras are usually device-dependent, i.e., the generated color information (usually presented in RGB color space) is dependent on the imaging characteristics of specific cameras. This is a serious problem in computer-aided tongue image analysis because it relies on the accurate rendering of color information. In this paper, we propose an optimized correction scheme that corrects the tongue images captured in different device-dependent color spaces to the target device-independent color space. The correction algorithm in this scheme is generated by comparing several popular correction algorithms, i.e., polynomial-based regression, ridge regression, support vector regression, and neural network mapping algorithms. We test the performance of the proposed scheme by computing the CIE L(*)a(*)b(*) color difference (∆E(ab)(*)) between estimated values and the target reference values. The experimental results on the colorchecker show that the color difference is less than 5 (∆E(ab)(*) < 5), while the experimental results on real tongue images show that the distorted tongue images (captured in various device-dependent color spaces) become more consistent with each other. In fact, the average color difference among them is greatly reduced by more than 95%.