Depression can be reflected by long-term human spatio-temporal facial behaviours. While human face videos recorded in real-world usually have long and variable lengths, existing video-based depression assessment approaches frequently re-sample/down-sample such videos to short and equal-length videos, or split each video into several equal-length segments, where segment-level spatio-temporal facial behaviours are suppressed as a vector-style representations for RNN-based long-term (video-level) modelling. Both strategies lead to crucial information loss and distortion. In this paper, we propose a novel graph-style data structure called Matrixial Graph and an effective Matrixial Graph Neural Network (MGNN) for face video-based depression assessment, which can directly and end-to-end model long-term depression-specific spatio-temporal facial cues from variable-length videos without resampling/splitting videos or suppressing video segments to vectors. Importantly, the nodes in our matrixial graph are capable of including matrices of different shapes, and thus nodes of a matrix graph can directly represent all frame-level 2D facial feature maps (or images themselves) of an entire video regardless of its length. Then, our MGNN is the first GNN that can jointly process matrixial graphs containing varying numbers of nodes, which further learns matrix-style edge features, thereby facilitating to explicit model video-level multi-scale spatio-temporal facial behaviours among matrixial graph nodes for depression assessment. Experiments show that the explicit spatio-temporal modeling on 2D facial feature maps, facilitated by our matrixial graph/MGNN, provided significant benefits, leading our approach to achieve new state-of-the-art performances on AVEC2013 and AVEC2014 datasets with large advantages.

Digital images captured by unstable imaging systems often simultaneously suffer from random noise and stripe noise. Due to the complex noise distribution, denoising and destriping methods based on simple handcrafted priors may leave residual noise. Although supervised methods have achieved some progress, they rely on large-scale noisy-clean image pairs, which are challenging to obtain in practice. To address these problems, we propose a self-supervised image denoising and destriping method based on blind-spot regularization, named Self-BSR. This method transforms the overall denoising and destriping problem into a modeling task for two spatially correlated signals: image and stripe. Specifically, blind-spot regularization leverages spatial continuity learned by the improved blind-spot network to separately constrain the reconstruction of image and stripe while suppressing pixel-wise independent noise. This regularization has two advantages: first, it is adaptively formulated based on implicit network priors, without any explicit parametric modeling of image and noise; second, it enables Self-BSR to learn denoising and destriping only from noisy images. In addition, we introduce the directional feature unshuffle in Self-BSR, which extracts multi-directional information to provide discriminative features for separating image from stripe. Furthermore, the feature-resampling refinement is proposed to improve the reconstruction ability of Self-BSR by resampling pixels with high spatial correlation in the receptive field. Extensive experiments on synthetic and real-world datasets demonstrate significant advantages of the proposed method over existing methods in denoising and destriping performance. The code will be publicly available at https://github.com/Jocobqc/Self-BSR.

Saliency object ranking (SOR) has attracted significant attention recently. Previous methods usually failed to ex-plicitly explore the saliency degree-related relationships between objects. In this paper, we propose a novel Domain Separation Graph Neural Network (DSGNN), which starts with separately extracting the shape and texture cues from each object, and builds an shape graph as well as a texture graph for all objects in the given image. Then, we propose a Shape-Texture Graph Domain Separation (STGDS) module to separate the task-relevant and irrelevant information of target objects by explicitly modelling the relationship between each pair of objects in terms of their shapes and textures, respectively. Furthermore, a Cross Image Graph Domain Separation (CIGDS) module is introduced to explore the saliency degree subspace that is robust to different scenes, aiming to create a unified representation for targets with the same saliency levels in different images. Importantly, our DSGNN automatically learns a multi-dimensional feature to represent each graph edge, allowing complex, diverse and ranking-related relationships to be modelled. Experimental results show that our DS-GNN achieved the new state-of-the-art performance on both ASSR and IRSR datasets, with large improvements of 5.2% and 4.1% SA-SOR, respectively. Our code is provided in https://github.com/Wu-ZJ/DSGNN.

Exploiting complementary advantages of different reflection and scattering properties of near-infrared (NIR) images and visible (VIS) images, this letter first proposes a defogging model for single image input and then develops an extended model, a fusion model for NIR and VIS color images, to enhance the visibility of image objects in scattering environments. Our fusion model enhances the extracted details of NIR and VIS images by filtering with our defogging model in the frequency domain that takes into account the energy preservation of these two types of images in addition to the high resolution of the fused results. Finally, based on the initial fusion, we propose a color retention mapping method to keep the fusion results free of color distortion. Experimental results demonstrate that our proposed method not only achieves good defogging effect, but also can effectively combine the complementary NIR and VIS information in image color and visibility.

Pixel-level semantic segmentation is a challenging task with a huge amount of computation, especially if the input sizes are large. In the segmentation network, apart from the pyramid backbone network, an extra decoder network is often employed to recover the spatial detail information. In this paper, we put forward a method for single-shot segmentation in a feature residual pyramid network (RPNet), which learns the coarse results and residuals of segmentations by decomposing the label at different levels of residual blocks. Specifically speaking, we use the residual features to learn the edges and details, and we also use the top-level feature to learn the coarse segmentation result. At the testing phase, the predicted residuals are used to enhance the details of the coarse segmentation result. Residual learning blocks split the network into several shallow sub-networks by level-wise training, which facilitates the gradient propagation in the RPNet. We then evaluate the proposed method and compare it with the recent state-of-the-art methods on CamVid and Cityscapes datasets. The proposed single-shot segmentation based on the RPNet achieves impressive results with high efficiency on the pixel-level segmentation task.

Spatially regularised discriminative correlation filters (SRDCFs) introduce spatial regularisation weights to mitigate the boundary effects caused by circular convolution which obtains superior performance. However, spatial regularisation is computationally expensive; this limits the real‐time performance of SRDCF. This study proposes high‐speed spatial constraint to DCFs (HSCDCFs) for tracking. Using a large area of the sample to learn a CF, then, the authors introduce the spatial constraint to penalise CF coefficients. Their method formulation allows the CFs to efficiently learn a mass of negative samples and high‐quality positive samples. They perform experiments on two benchmark datasets: OTB‐2013 and OTB‐2015. Compared to SRDCF, they provide a slightly reduce of 2.7 and 3.1%, respectively, in mean overlap precision, their method obtains the real‐time speed of 62.5 fps which is ten times faster than SRDCF.

In view of the shortcoming of traditional image fusion based on discrete wavelet transform (DWT) with unclear textural information, an effective visible light and infrared image fusion algorithm via feature residual and statistical matching is proposed in this study. First, the source images are decomposed into low‐frequency coefficients and high‐frequency coefficients by DWT. Second, two different fusion schemes are designed for the low‐frequency coefficients and high frequency ones, respectively. The low‐frequency coefficients are fused by a local feature residual‐based scheme to achieve adaptive fusion; the high‐frequency coefficients are accomplished by a local statistical matching‐based scheme to extract the edge information effectively. Finally, the fused image is obtained by inverse DWT. Experimental results demonstrate that the proposed method can produce a more accurate fused image, leading to an improved performance compared with existing methods.

Sparse and redundant representations perform well in image denoising. However, sparsity-based methods fail to denoise low-light-level (LLL) images because of heavy and complex noise. They consider sparsity on image patches independently and tend to lose the texture structures. To suppress noises and maintain textures simultaneously, it is necessary to embed noise invariant features into the sparse decomposition process. We, therefore, used a local structure preserving sparse coding (LSPSc) formulation to explore the local sparse structures (both the sparsity and local structure) in image. It was found that, with the introduction of spatial local structure constraint into the general sparse coding algorithm, LSPSc could improve the robustness of sparse representation for patches in serious noise. We further used a kernel LSPSc (K-LSPSc) formulation, which extends LSPSc into the kernel space to weaken the influence of linear structure constraint in nonlinear data. Based on the robust LSPSc and K-LSPSc algorithms, we constructed a local sparse structure denoising (LSSD) model for LLL images, which was demonstrated to give high performance in the natural LLL images denoising, indicating that both the LSPSc- and K-LSPSc-based LSSD models have the stable property of noise inhibition and texture details preservation.

The stimulus response of the classical receptive field (CRF) of neuron in primary visual cortex is affected by its periphery [i.e., non-CRF (nCRF)]. This modulation exerts inhibition, which depends primarily on the correlation of both visual stimulations. The theory of periphery and center interaction with visual characteristics can be applied in night vision information processing. In this paper, a weighted kernel principal component analysis (WKPCA) degree of homogeneity (DH) amended inhibition model inspired by visual perceptual mechanisms is proposed to extract salient contour from complex natural scene in low-light-level image. The core idea is that multifeature analysis can recognize the homogeneity in modulation coverage effectively. Computationally, a novel WKPCA algorithm is presented to eliminate outliers and anomalous distribution in CRF and accomplish principal component analysis precisely. On this basis, a new concept and computational procedure for DH is defined to evaluate the dissimilarity between periphery and center comprehensively. Through amending the inhibition from nCRF to CRF by DH, our model can reduce the interference of noises, suppress details, and textures in homogeneous regions accurately. It helps to further avoid mutual suppression among inhomogeneous regions and contour elements. This paper provides an improved computational visual model with high-performance for contour detection from cluttered natural scene in night vision image.