As AI models are increasingly integrated into applications involving human interaction, understanding the alignment between human perception and machine vision has become essential. One example is the estimation of visual motion (optical flow) in dynamic applications such as driving assistance. While there are numerous optical flow datasets and benchmarks with ground truth information, human-perceived flow in natural scenes remains underexplored. We introduce HuPerFlow—a benchmark for human-perceived flow, measured at 2,400 locations across ten optical flow datasets, with ∼38,400 response vectors collected through online psychophysical experiments. Our data demonstrate that human-perceived flow aligns with ground truth in spatiotemporally smooth locations while also showing systematic errors influenced by various environmental properties. Additionally, we evaluated several optical flow algorithms against human-perceived flow, uncovering both similarities and unique aspects of human perception in complex natural scenes. HuPerFlow is the first large-scale human-perceived flow benchmark for alignment between computer vision models and human perception, as well as for scientific exploration of human motion perception in natural scenes. The HuPerFlow benchmark is publicly available on the HuPerFlow website.

Perceptual organization in the human visual system involves neural mechanisms that spatially group and segment image areas based on local feature similarities, such as the temporal correlation of luminance changes. Successful segmentation models in computer vision, including graph-based algorithms and vision transformer, leverage similarity computations across all elements in an image, suggest that effective similarity-based grouping should rely on a global computational process. However, whether human vision employs a similarly global computation remains unclear due to the absence of appropriate methods for manipulating similarity matrices across multiple elements within a stimulus. To investigate how "temporal similarity structures" influence human visual segmentation, we developed a stimulus generation algorithm based on Vision Transformer. This algorithm independently controls within-area and cross-area similarities by adjusting the temporal correlation of luminance, color, and spatial phase attributes. To assess human segmentation performance with these generated texture stimuli, participants completed a temporal two-alternative forced-choice task, identifying which of two intervals contained a segmentable texture. The results showed that segmentation performance is significantly influenced by the configuration of both within- and cross-correlation across the elements, regardless of attribute type. Furthermore, human performance is closely aligned with predictions from a graph-based computational model, suggesting that human texture segmentation can be approximated by a global computational process that optimally integrates pairwise similarities across multiple elements.

Visual motion processing is essential for humans to perceive and interact with dynamic environments. Despite extensive research in cognitive neuroscience, image-computable models that can extract informative motion flow from natural scenes in a manner consistent with human visual processing have yet to be established. Meanwhile, recent advancements in computer vision (CV), propelled by deep learning, have led to significant progress in optical flow estimation, a task closely related to motion perception. Here we propose an image-computable model of human motion perception by bridging the gap between biological and CV models. Specifically, we introduce a novel two-stages approach that combines trainable motion energy sensing with a recurrent self-attention network for adaptive motion integration and segregation. This model architecture aims to capture the computations in V1-MT, the core structure for motion perception in the biological visual system, while providing the ability to derive informative motion flow for a wide range of stimuli, including complex natural scenes. In silico neurophysiology reveals that our model's unit responses are similar to mammalian neural recordings regarding motion pooling and speed tuning. The proposed model can also replicate human responses to a range of stimuli examined in past psychophysical studies. The experimental results on the Sintel benchmark demonstrate that our model predicts human responses better than the ground truth, whereas the state-of-the-art CV models show the opposite. Our study provides a computational architecture consistent with human visual motion processing, although the physiological correspondence may not be exact.

Most structures of deep neural networks (DNN) are with a fixed complexity of both computational cost (parameters and FLOPs) and the expressiveness. In this work, we experimentally investigate the effectiveness of using neural ordinary differential equations (NODEs) as a component to provide further depth to relatively shallower networks rather than stacked layers (depth) which achieved improvement with fewer parameters. Moreover, we construct deep neural networks with flexible complexity based on NODEs which enables the system to adjust its complexity while training. The proposed method achieved more parameter-efficient performance than stacking standard DNNs, and it alleviates the defect of the heavy cost required by NODEs.

Semantic segmentation is a fundamental task in computer vision and image processing. Although existing methods based on the fully convolutional network (FCN) have greatly improved the accuracy, it still does not show satisfactory results on tiny objects and boundary regions. One of the problems is that the current FCN-based methods ignore details such as the image’s contours and edges because of over downsampling operations in the CNN encoder backbone. In signal processing, excessive down-sampling will incur spectrum aliasing, thus losing high-frequency details. This work presents a high-frequency graph convolution operation to solve the above problems. Traditional image processing generally uses the high-pass filter to extract image contours. We accordingly suppose that the high-frequency information is vital for the extractions of boundary clues and details. We implement our strategy and conduct experiments on the Cityscapes dataset, which demonstrate the effectiveness of our high-frequency graph convolution block on semantic segmentation. The proposed method achieves comparable performance and dramatically improves the performance of small objects like the rider, traffic signs, etc.

Deep neural networks (DNNs) have extensively promoted data generation development; the quality of these generated content has achieved an impressive new level. Therefore, manipulated content, especially facial manipulation, is a growing concern for online information legitimacy. Most current deep learning-based methods depend on local features sampled by convolutional kernels and lack knowledge globally. To address the problem, we propose a dual-path pipeline using Neural Ordinary Differential Equations (NODE) based neural network and facial-feature biased transformer to deal with the visual content from a different view. The transformer path could link these landmarks in a long-range, moreover, we adopt an attention guided augmentation based self-ensemble for more robust performance. Extensive experiments show that our system could surpass several commonly used approaches in terms of video-level accuracy and AUC with better interpretability.

Semantic segmentation requires both a large receptive field and accurate spatial information. Although existing methods based on a fully convolutional network have greatly improved the accuracy, the prediction results still do not show satisfactory when parsing small objects and boundary regions. We propose a refinement algorithm to improve the result generated by the front network. Our method takes a modified double-branches network to generate both segmentation masks and semantic boundaries, which serve as refinement algorithms' input. We creatively introduce information entropy to represent the confidence of the neural network's prediction corresponding to each pixel. The information entropy combined with the semantic boundary can capture those unpredictable pixels with low-confidence through Monte Carlo sampling. Each selected pixel will serve as the initial seed for directed local search and refinement. According to the initial seed, our purpose is tantamount to searching the neighbor high-confidence regions, and the re-labeling approach is based on high-confidence results. Remarkably, our method adopts a directed regional search strategy based on gradient descent to find the high-confidence region effectively. Our method can be flexibly embedded into the existing encoder backbone at a trivial computational cost. Our refinement algorithm can further improve the state of the art method's accuracy both on Cityscapes and PASCAL VOC datasets. In evaluating some small objects, our method surpasses most of the state of the art methods.

Image compression is a basic task in image processing. The existing methods always have problems such as the loss of image details and the reconstructed image does not conform to human vision. This paper presents an adaptive image compression algorithm that relies on GAN based semantic-perceptual residual compensation, which is available to offer visually pleasing reconstruction at a low bitrate. Our method derive from a U-shaped encoder-decoder structure accompanied by a well-designed dense residual connection with a strip pooling module to improve the original auto-encoder. Besides, we utilize the idea of adversarial learning by introducing a discriminator, thus constructed a complete GAN. To improve the coding efficiency, we creatively designed an adaptive semantic-perception residual compensation block based on the Grad-CAM algorithm. Through the strategy of adversarial learning, the reconstructed image is more towards the distribution of the real image, and further semantic perception can achieve higher quality compression of the region of interest from the human attention. Besides, we combine multiple existing quantitative methods, including the latest FLIF lossless compression algorithm, BPG vector compression algorithm and soft-quantization to perform deeper compression on the image. Experimental results, including PSNR, MS-SSIM demonstrate that the proposed approach outperforms the current state-of-the-art image compression methods.