Light field image depth estimation methods involve a large number of parameters and floating-point operations. This makes FPGA-based acceleration design a huge challenge, especially when pursuing high-precision acceleration design on resource-limited FPGAs. Motivated by this issue, a resource-efficient hardware accelerator based on a high-precision, low-bit lightweight light field image depth estimation network scheme is proposed, named RE-LFDE. First, we present a parameter-sharing, low-bit and lightweight network. It is able to improve accuracy, simplify the network structure and reduce network parameters. Secondly, we design a time-division multiplexing hardware-software co-design dataflow structure and build a resource-efficient acceleration engine, which can be deployed on the resource-limited FPGAs efficiently. Experimental results show that the average MSE on the 4D LF benchmark of RE-LFDE and its full-precision networks can be reduced to 3.265 and 3.363, respectively, while the weight parameters can be as low as 0.30MB and 0.04MB. Furthermore, on the ZCU104 platform, the consumption of BRAM and LUTRAM can be reduced to 22.44% and 13.64%, respectively. The code and model of the proposed method are available at https://github.com/sansi-zhang/RE-LFDE .

Facial expressions are a powerful medium for conveying emotions. In facial expression recognition (FER) field, the difficulty of collecting specific expressions often leads to class imbalance in mainstream datasets, significantly reducing the classification accuracy of deep neural networks. To address these issues, we propose a stable-diffusion-based augmentation method for facial expression (SDAFE) that resolves class imbalance problems and enhances data generation quality through cross-modal label guidance. By leveraging the neutrality of neutral faces, we generate additional expressions to balance the dataset classes. We introduce a peak signal-to-noise ratio (PSNR) filter to ensure the high quality of the generated images and a cosine similarity cross-modal filter based on CLIP encoders to ensure that the content of the generated images accurately aligns with their labels. Furthermore, we introduce a novel model, FERNeXt, which demonstrates outstanding performance in FER tasks, surpassing the state-of-the-art accuracy on the FER2013 dataset and achieving strong results on the RAF-DB and NHFI datasets. Subsequently, the performance of several models across different datasets significantly improves through the use of SDAFE in our experiments.

Reflections on the water surface hinder the extraction of valuable information from water surface images. To remove reflections from water surface images, we construct a synthetic dataset and propose a multi-task network for water surface reflection detection and removal. Specifically, we first use a U-Net-based reflection detection module to generate a reflection mask, followed by a GAN-based network to remove the reflection. To extract multi-level features from the images, we design a color feature extraction network and a detail feature extraction network. Finally, to enhance the model's ability to remove large-area reflections, we pre-train the reflection removal network on an image restoration dataset. Experimental results on the proposed synthetic dataset and real water surface reflection images from the Internet show that our method significantly outperforms other methods in water surface reflection detection and removal.

With the rapid advancements in object detection, oriented object detection has gained increasing attention. However, challenges such as boundary discontinuity and square-like problems in oriented object detection persist, as most existing methods directly regress the rotation angle, leading to instability in boundary angle prediction. To address these challenges, this paper introduces a novel rotation angle encoding method called the Composite Trigonometric Function Coder (CTFC), which transforms discrete angles into continuous curves. Leveraging the smooth characteristics of trigonometric functions, CTFC eliminates abrupt curvature changes, thereby avoiding the sudden angle steps inherent in conventional methods and simplifying the optimization process. Experiments conducted on three datasets validate the effectiveness of the proposed method. Additionally, the performance of CTFC has been further analyzed using three different detector heads. Experimental results and data analysis demonstrate that CTFC is efficient and effective in oriented object detection.

3D Gaussian Splatting (3DGS) is highly effective at synthesizing new views but could suffer from underfitting, especially when oversized Gaussians fail to capture fine details. To address this, we propose Visibility-GS, a novel approach that wisely adjusts Gaussians’ gradient magnitudes based on Gaussian Visibility during the Adaptive Density Control (ADC) process. Here, Gaussian visibility refers to how well a Gaussian is observed from each view. By considering visibility, our method corrects the Visibility Bias in 3DGS and introduces a Gaussian visibility-based densification mechanism. It dynamically guides oversized Gaussians to split, while leaving the splitting condition of other Gaussians unchanged. Extensive qualitative and quantitative experiments demonstrate that our approach consistently improves quality across a range of benchmarks, without sacrificing efficiency.

Video anomaly detection (VAD) methods are mostly CNN-based or Transformer-based, achieving impressive results, but the focus on detection accuracy often comes at the expense of inference speed. The emergence of state space models in computer vision, exemplified by the Mamba model, demonstrates improved computational efficiency through selective scans and showcases the great potential for long-range modeling. Our study pioneers the application of Mamba to VAD, dubbed VADMamba, which is based on multi-task learning for frame prediction and optical flow reconstruction. Specifically, we propose the VQ-Mamba Unet (VQ-MaU) framework, which incorporates a Vector Quantization (VQ) layer and Mamba-based Non-negative Visual State Space (NVSS) block. Furthermore, two individual VQ-MaU networks separately predict frames and reconstruct corresponding optical flows, further boosting accuracy through a clip-level fusion evaluation strategy. Experimental results validate the efficacy of the proposed VADMamba across three benchmark datasets, demonstrating superior performance in inference speed compared to previous work. Code is available at https://github.com/jLooo/VADMamba.

The 3-D vision computing is a key application in unmanned systems, satellites, and planetary rovers. Learning-based light field (LF) depth estimation is one of the major research directions in 3-D vision computing. However, conventional learning-based depth estimation methods involve a large number of parameters and floating-point operations, making it challenging to achieve low-power, fast, and high-precision LF depth estimation on a field-programmable gate array (FPGA). Motivated by this issue, an FPGA-based low-bit, lightweight LF depth estimation network (L$^{3}\text {FNet}$) is proposed. First, a hardware-friendly network is designed, which has small weight parameters, low computational load, and a simple network architecture with minor accuracy loss. Second, we apply efficient hardware unit design and software-hardware collaborative dataflow architecture to construct an FPGA-based fast, low-bit acceleration engine. Experimental results show that compared with the state-of-the-art works with lower mean-square error (mse), L$^{3}\text {FNet}$can reduce the computational load by more than 109 times and weight parameters by approximately 78 times. Moreover, on the ZCU104 platform, it requires 95.65% lookup tables (LUTs), 80.67% digital signal processors (DSPs), 80.93% BlockRAM (BRAM), 58.52% LUTRAM, and 9.493-W power consumption to achieve an efficient acceleration engine with a latency as low as 272 ns. The code and model of the proposed method are available athttps://github.com/sansi-zhang/L3FNet.

Abstract Rain can severely hamper the visibility of scene objects. Although existing deep learning methods have reported promising performance, they often fail to obtain satisfactory results in many practical situations, especially when the input image contains both rain streaks and haze‐like degradation. In this paper, a new two‐stage method based on attention smoothed dilated network (SDN) is proposed. Unlike most fully‐supervised methods, the mixture of rain streaks and haze‐like effects is considered in the model. The proposed method consists of two stages. First, a generative adversarial network guided by the rain‐streak attention map is proposed to remove rain streaks, where a multi‐stage attention module is used to accurately locate rain streaks in the generator. Second, haze‐like effects are further removed through SDN with the same structure as the generator. Extensive experiments on multiple datasets show that the method outperforms the state‐of‐the‐art in both objective evaluation and visual quality.

Abstract Deraining has been attracting a lot of attention from researchers, and various methods have been proposed, especially deep‐networks are widely adopted in recent years. Their structures and learning become more and more complicated and diverse, making it difficult to analyze the contributions and improvements. In this paper, a comprehensive review for current rain removal methods is first provided to show their contributions. Specifically, they are reviewed in terms of handing rain streaks and rain mist. Second, besides evaluating their rain removal ability, they are also evaluated in terms of their impact on subsequent stereo‐matching task. To this end, a new deraining dataset is first prepared, called Rain‐Kitti2012 and Rain‐Kitti2015. They are created by adding rain part to clean image‐pairs in Kitti2012 and Kitti2015. By then, nine state‐of‐the‐art deraining methods are evaluated with full‐reference and no‐reference image quality assessment metrics. Furthermore, the blurriness and distortion types introduced during deraining are measured. Finally, three learning‐based stereo matching methods are compared, and they take the outputs of deraining methods as inputs. It is further discussed how derained images influence the accuracy of stereo matching, which can provide some insight for jointly handling rain removal and stereo matching. 1: A comprehensive review for the current rain removal methods is provided. They are categorized into rain‐streak‐oriented and rain‐mist‐oriented approaches in terms of degradation type, and are categorized into model‐driven and data‐driven approaches in terms of methodology. 2: A new image deraining dataset is introduced, which is the first dataset that can be used to perform stereo‐matching‐driven evaluation for deraining methods. The dataset is created by adding rain part to clean images in KITTI2012 and KITTI2015. 3: We evaluate 9 deep learning based deraining methods with full‐reference and no‐ reference metrics. In addition, the types of distortions produced by these methods are discussed and measured quantitatively. And, the impact of 9 deraining methods on the subsequent stereo matching task is evaluated, which can provide some insight on how to design stereo matching task‐driven deraining methods.

Abstract Single image de‐raining based on convolutional neural network (CNN) has made considerable progress in recent years. However, usually the de‐rained result has dark artifacts and image textures tend to be over‐smoothed. In this paper, a pyramid non‐local enhanced residual dense network is proposed to reduce such distortion. Firstly, the down‐sampled images are input into the Laplacian pyramid, which can extract the overall and partial texture clues, and subsequently a set of images of different scales are produced. Secondly, these images are fed into a non‐local enhanced residual dense block, which can not only capture long‐distance dependencies of feature maps, but also fully utilizes the hierarchical features in every dense block, leading to high accuracy of rain streaks extraction and better preservation of image edge detail. Finally, the de‐rained image is gradually restored by Gaussian reconstruction pyramid. Experimental results on both synthetic data and real‐world data show that the artifacts distortion is obviously reduced by the proposed network. And the quality of de‐rained image is significantly improved compared with the state‐of‐the‐art methods.

Abstract Salient target detection in hyperspectral image is a significant task in image segmentation, target tracking, image classification and so on. Many existing saliency detection algorithms for hyperspectral image detection cannot present the boundary of the salient target well and the description of the target is not enough. A method based on visual attention to detect the salient target of hyperspectral image is proposed in this paper. In this method, frequency‐tuned (FT) salient detection model is combined with spectral salient to detect target in hyperspectral image. FT model is used to get target with clear border, and spectral information is made full use of to improve the accuracy of target detection. Firstly, FT is used to detect saliency of hyperspectral image and the saliency map is generated. Then, spectral information of the hyperspectral image is measured by similarity, and the spectral saliency is obtained by calculating spectral angle distance between the spectral vectors. Finally, the FT's saliency map and the spectral saliency map are combined to form the final saliency target maps. Experimental results show that our method is superior to other methods in saliency target detection of hyperspectral image, and the precision‐recall curve and F‐measure are better as well.

Neuron detection is a key step in individualizing and counting neurons which are important for assessing physiological and pathophysiological information. A large number of methods including deep learning networks have been proposed but mainly targeting regions with few aggregated neurons. The objective of this paper is to address an automated neuron detection problem in heterogeneous hippocampus region with different degrees of neuron aggregation. Since deep learning networks require a lot of ground truths but neuron instance annotation is impossible in regions where numerous neurons are clustered, ground truth of centroids marked at the center of neurons is created for training. We propose a multiscale convolutional neural network (CNN) to regress neuron centroid mapping across image. Using multiscale information makes the proposed network applicable not only for single individual neurons, but also for a large number of aggregated neurons. Experimental results show that our method is superior to state-of-the-art deep learning-based algorithms. To our knowledge, this is the first deep learning study to detect neurons in regions of highly clustered neurons.

The seed-and-expand scheme is appropriate for multiple view stereo, since it can build dense point clouds adaptively by avoiding unnecessary computation. However, due to the irregularity of the algorithm, it is not suitable for parallel computing on general public utilities (GPU). This paper is the first attempt to implement the irregular seed-and-expand method on GPU for multiple view stereo problems. Meanwhile, a hierarchical parallel computing architecture is also proposed to maximize the usage of both CPU and GPU. The adaptivity of the seed-and-expand scheme is pushed further by processing a pixel several rounds while, in order to maintain regularity for GPU implementation, every seed has exactly the same behavior in a single round of optimization. The high adaptivity also improves the robustness of the proposed method, thus aggressive matching score and a view selection method can be used to improve the reconstruction completeness extensively, without smearing out local details and lowering the accuracy. Compared with the state of the art, the proposed method achieves higher accuracy and completeness on standard datasets. The proposed method is also very fast. It is maximally five times faster than other methods running on a CPU and is on par with the regular depth map-based methods on GPU, which are naturally suitable for GPU acceleration.

When using fixed-window phase correlation (PC) to estimate the disparity of stereo images, the precision is usually rather poor due to large depth differences of scenes and noise, and this problem is specially severe when using unmanned aerial vehicle (UAV) image pairs to extract the digital elevation model of mountain land. To tackle this problem, this paper proposes a hierarchical and adaptive PC, which includes three steps: First, PC with the initialized window is performed to coarsely estimate a disparity value, along with the peak of the Dirichlet function for each pixel; then, an additional round of PC is performed for each pixel using the window of smaller size and with being guided by the coarsely estimated disparity; finally, the previous two steps are iteratively performed until convergence. In particular, using the peak of the Dirichlet function of each pixel in step two, we can drop out the influence of dramatically changing areas such as river; moreover, the scheme can minimize the influence of boundary overreach. The novel scheme has been tested on a large number of UAV images captured at mountainous regions in southwest China, showing that the proposed method is superior to the state-of-the-art methods, especially in handling UAV images of the high mountains and rivers.