Diffusion models have become a rising star in image super-resolution (SR) tasks. However, it is not trivial to apply diffusion models for light field (LF) image SR, which requires maintaining the high-quality visual appearance of each sub-aperture image (SAI) and the angular consistency between the different SAIs. This paper proposes the first diffusion-based LF image SR model, namely LFSRDiff, by incorporating the LF disentanglement mechanism and residual modeling. Specifically, we introduce a disentangled U-Net (Distg U-Net) for diffusion models, enabling improved extraction and fusion of the spatial and angular information in LF images. Furthermore, we leverage residual modeling in diffusion to learn the residual between the upsampled low-resolution and the ground truth high-resolution, which significantly accelerates model training and yields superior results compared to direct learning. Extensive experiments conducted on the five datasets demonstrate the effectiveness of our approach, which can produce realistic SR results and achieve the highest perceptual metric in terms of LPIPS. Code is publicly available at https://github.com/chaowentao/LFSRDiff.

Data augmentation (DA) is an effective approach for enhancing model performance with limited data, such as light field (LF) image super-resolution (SR). LF images inherently possess rich spatial and angular information. Nonetheless, there is a scarcity of DA methodologies explicitly tailored for LF images, and existing works tend to concentrate solely on either the spatial or angular domain. This paper proposes a novel spatial and angular DA strategy named MaskBlur for LF image SR by concurrently addressing spatial and angular aspects. MaskBlur consists of spatial blur and angular dropout two components. Spatial blur is governed by a spatial mask, which controls where pixels are blurred, i.e., pasting pixels between the low-resolution and high-resolution domains. The angular mask is responsible for angular dropout, i.e., selecting which views to perform the spatial blur operation. By doing so, MaskBlur enables the model to treat pixels differently in the spatial and angular domains when super-resolving LF images rather than blindly treating all pixels equally. Extensive experiments demonstrate the efficacy of MaskBlur in significantly enhancing the performance of existing SR methods. We further extend MaskBlur to other LF image tasks such as denoising, deblurring, low-light enhancement, and real-world SR.

A single light field image, containing multiple views, can be utilized in conjunction with estimated depth map to generate three-dimensional (3D) point cloud. It is noteworthy that many depth estimation algorithms frequently produce depth maps plagued by issues such as holes and blurred edges. Many algorithms use global optimization directly to optimize depth maps without considering the effects of holes and blurred edges, resulting in the loss of the sharp features of the point cloud. Different from their methods, we consider both holes and blurry edges as outlier points. Then, we classify angular sampling images based on color consistency differences and employ interclass distances to detect outlier points. With the help of the outlier mask, we can obtain the depth confidence for each point. Combining with the depth confidence, we define a weighting coefficient to determine the optimization neighborhood of the outlier point. Finally, the point cloud generated by the depth map can be further optimized based on the outlier mask. Experimental results on synthetic and real datasets demonstrate the effectiveness of the proposed algorithm.

Bottom-up human pose estimation has raised more investigation in recent years, especially 2D keypoints regression. However, the state-of-art DEKR [1] still has some aspects (e.g., speed and accuracy) to be improved. In this paper, we propose a new framework named DEKRv2, which has been enhanced compared to DEKR. When DEKR calculates the offset of each keypoint, it only considers the features of the current keypoint and neglects the constraints between the adjacent keypoints. We adopt a coarse-to-fine feature extraction method to obtain a more accurate feature location of keypoints for this problem. We also find that the multibranch network in DEKR is very time-consuming because it is serial. We designed a more effective module based on Group Convolution to replace the multi-branches network in DEKR, and it can reduce reasoning time. Experiments on the CrowdPose dataset show that our method achieves superior compared with DEKR in speed or accuracy, respectively. In the single-scale test, our method obtains 66.6 mAP, 0.6 higher than DEKR. The codes and models are available at https://github.com/chaowentao/DEKRv2.

Face sketch-photo synthesis has important usage in law enforcement and human authentication. Due to the sparse information (no color or texture), the abstraction level, the diversity of sketches, and the domain gap between sketch and photo, it is challenging to synthesize a photo-realistic photo from an input sketch. Moreover, the deficiency of data also restricts the synthesis performance. In this paper, we present a high-fidelity face sketch-photo synthesis method using Generation Adversarial Network (GAN). Our network adopts a deep residual U-Net as generator and a Patch-GAN with residual blocks as discriminator. We design effective loss functions by enforcing pixels, edges and high-level features of the produced face photos. Moreover, we augment the CUHK sketch dataset using an effective sampling method. With the improved GAN and augmented dataset, we achieve high-fidelity face photos. Qualitative and quantitative experiments demonstrate the approach outperforms other method. Further experiments with a sketch-based photo editing application also validate the performance of our method.