3D object detection has achieved significant progress in outdoor LiDAR point clouds, however, the inherent irregularity and varying sparsity distribution of point occupancy present a key challenge. Existing transformer-based 3D detectors often treat all tokens within the attention window as equally important, regardless of varying sparsity, which not only fails to address the disparities between the varying beam densities but also results in increased memory and computational costs. In this work, we propose an adaptive structure-aware cascaded transformer (ASCFormer) that dynamically captures density-insensitive multiscale structure features to model long-range dependencies via cascaded learning. Our ASCFormer detector includes an adaptive structure-aware token learning module that embeds voxel-level foreground probability and grid-level local density into the grid tokens to enhance structural perception capability. Moreover, we integrate these factors to compute significance scores, which are then utilized in inverse transform sampling to select a subset of multiscale tokens with varying receptive field sizes. To improve the training convergence of the window-based transformer in 3D voxel space, we employ cascaded learning via cross-stage attention to enhance the feature representation capability and refine the localization precision of 3D bounding boxes. This design of structure-aware reweighting effectively enhances the cascade paradigm, making to more adaptable to the varying sparsity distribution of point clouds. Extensive experiments on the KITTI and Waymo Open datasets demonstrate that the proposed ASCFormer detector achieves exceptional performance compared with state-of-the-art 3D object detection methods. The source code is publicly available at https://github.com/Xinglong-Li1/ASCFormer.

Low-light video enhancement is highly demanding in maintaining spatiotemporal color consistency. Therefore, improving the accuracy of color mapping and keeping the latency low are challenging. On this basis, we propose incorporating wavelet-priori for the 4D lookup table (WaveLUT), which effectively enhances the color coherence between video frames and the accuracy of color mapping while maintaining low latency. Specifically, we use the wavelet low-frequency domain to construct an optimized lookup prior and achieve an adaptive enhancement effect through a designed wavelet-prior 4D lookup table. To effectively compensate for the a priori loss in the low light region, we further explore a dynamic fusion strategy that adaptively determines the spatial weights on the basis of the correlation between the wavelet lighting prior and the target intensity structure. In addition, during the training phase, we devise a Fourier-text driven appearance reconstruction method that dynamically balances brightness and content through multimodal semantics-driven Fourier spectra. Extensive experiments on a wide range of benchmark datasets show that this method effectively enhances the previous method's ability to perceive the color space and achieves metric-favourable and perceptually oriented real-time enhancement while maintaining high efficiency. The code is available athttps://github.com/hejh8/WaveLUT.

With the advancement of machine vision technology, automated vision inspection systems are needed in broad quality control scenarios. This article proposes a neighborhood attention-based feature reconstruction method for image anomaly detection and localization (NAFRAD). To address the challenges of data scarcity, low visibility, and irregular defect shapes in unsupervised anomaly detection, we introduce a feature reconstruction framework that preserves high-level abstract features rather than focusing on pixel-level reconstruction. This approach enhances model robustness and generalizability by leveraging neighborhood attention (NA) mechanisms, which simultaneously capture local details and the global context through a sliding window strategy. The NA-based autoencoder reconstructs normal features by aggregating local inductive biases with translational equivariance, enabling precise anomaly localization. Extensive experiments on the MVTec Anomaly Detection (MVTec AD) dataset—comprising 15 categories with 5,354 images—demonstrate the superiority of NAFRAD. It achieves state-of-the-art performance with AUROC \({}_{I}\) = 99.02, AUROC \({}_{P}\) = 98.99, and AP = 79.40, outperforming existing methods by 3.6% in AP and 0.89% in AUROC \({}_{P}\) . The framework’s effectiveness is validated through ablation studies, visualization of feature reconstruction, and comparisons with eight leading unsupervised methods. The code is made public at https://github.com/Math-Computer/NAFRAD .

Image dehazing remains a challenging task in real-world scenarios. Unlike synthetic datasets, real-world hazy images often exhibit nonuniform atmospheric degradation, severe haze accumulation, and significant texture loss, making it difficult for existing methods to restore realistic appearances and fine details consistently. To address these challenges, we propose a frequency-aware learning-based unpaired image dehazing network for real-world hazy scenes. First, we propose a content-aware state space modeling paradigm, focus on details through high-frequency enhancement, and incorporate global contextual understanding at the encoding stage, enabling adaptive representation of complex textures. Second, to effectively handle spatially nonuniform degradation, a hazy density estimation module is designed to guide multiple expert-gated feedback units, which dynamically select feature fusion paths. Finally, we propose a contour-guided differentiable frequency domain enhancement mechanism to explicitly recover edge and texture details in degraded regions. Extensive experiments on real-world hazy datasets under unsupervised settings demonstrate that our method achieves competitive performance, validating its effectiveness and strong practical potential under complex atmospheric conditions. The code is available at https://github.com/Fan-pixel/FAL-Net .

Existing full-reference image quality assessment (FR-IQA) methods often fail to capture the complex causal mechanisms that underlie human perceptual responses to image distortions, limiting their ability to generalize across diverse scenarios. In this paper, we propose an FR-IQA method based on abductive counterfactual inference to investigate the causal relationships between deep network features and perceptual distortions. First, we explore the causal effects of deep features on perception and integrate causal reasoning with feature comparison, constructing a model that effectively handles complex distortion types across different IQA scenarios. Second, the analysis of the perceptual causal correlations of our proposed method is independent of the backbone architecture and thus can be applied to a variety of deep networks. Through abductive counterfactual experiments, we validate the proposed causal relationships, confirming the model’s superior perceptual relevance and interpretability of quality scores. The experimental results demonstrate the robustness and effectiveness of the method, providing competitive quality predictions across multiple benchmarks. The source code is available at https://anonymous.4open.science/r/DeepCausalQuality-25BC.

Large vision-language models (VLMs) have significantly advanced human-object interaction (HOI) detection. However, existing VLM-based HOI detectors primarily rely on simple text prompt paradigms, specifically in relation to knowledge hallucination, with limited exploration of the intrinsic attributes or extrinsic context. In this paper, we propose a knowledge distilled group prompts learning method for HOI detection, termed GPL-HOI, which transfer knowledge from vision-language models via group prompts and knowledge distillation. Specifically, we design visual-textual group prompts by combining scene-aware, region-aware, and pose-aware prompt to guide knowledge transfer from VLMs. Additionally, we introduce a cross-modal group distillation module,which aligns the semantic features of both the vision and text models via KL divergence, encouraging the visual encoder to generate similar probability distributions to the text encoder through the learnable prompts. Extensive experiments demonstrate that our method surpasses state-of-the-art approaches in both conventional and zero-shot settings, achieving improvements of +2.04 mAP and +1.84 mAP on HICO-DET, respectively. Code will be available at https://github.com/hxqstree/GPL-HOI.

Low-light image enhancement aims to improve the visibility of degraded images to better align with human visual perception. While diffusion-based methods have shown promising performance due to their strong generative capabilities. However, their unidirectional modelling of degradation often struggles to capture the complexity of real-world degradation patterns, leading to structural inconsistencies and pixel misalignments. To address these challenges, we propose a bidirectional diffusion optimization mechanism that jointly models the degradation processes of both low-light and normal-light images, enabling more precise degradation parameter matching and enhancing generation quality. Specifically, we perform bidirectional diffusion-from low-to-normal light and from normal-to-low light during training and introduce an adaptive feature interaction block (AFI) to refine feature representation. By leveraging the complementarity between these two paths, our approach imposes an implicit symmetry constraint on illumination attenuation and noise distribution, facilitating consistent degradation learning and improving the model's ability to perceive illumination and detail degradation. Additionally, we design a reflection-aware correction module (RACM) to guide color restoration post-denoising and suppress overexposed regions, ensuring content consistency and generating high-quality images that align with human visual perception. Extensive experiments on multiple benchmark datasets demonstrate that our method outperforms state-of-the-art methods in both quantitative and qualitative evaluations while generalizing effectively to diverse degradation scenarios.Code

Effective modelling of uncertain information is crucial for quantifying uncertainty. Dempster–Shafer evidence (DSE) theory is a widely recognized approach for handling uncertain information. However, current methods often neglect the inherent a priori information within data during modelling, and imbalanced data lead to insufficient attention to key information in the model. To address these limitations, this paper presents a dynamic learning strategy based on nonuniform splitting mechanism and Hilbert space mapping. First, the framework uses a nonuniform splitting mechanism to dynamically adjust the weights of data subsets and combines the diffusion factor to effectively incorporate the data a priori information, thereby flexibly addressing uncertainty and conflict. Second, the conflict in the information fusion process is reduced by Hilbert space mapping. Experimental results on multiple tasks show that the proposed method significantly outperforms state-of-the-art methods and effectively improves the performance of classification and low-light image enhancement (LLIE) tasks. The code is available at https://anonymous.4open.science/r/Third-ED16.

Efficient Image Super Resolution (EISR) techniques are critical to meeting the real-time demands of resource-constrained devices. Current approaches focus too much on parameter reduction, thus slightly sacrificing image quality. To alleviate this issue, we propose a novel Hybrid Attention Dense Network (HADN) in this paper. HADN fully explores how to balance model performance and image recovery quality better. It incorporates two key designs: (1) We apply Hybrid-domain Attention Dense Blocks (HADB) with different feature layers cumulatively connected to enhance the representation of shallow features. (2) We designed a novel Hybrid-domain Attention Block (HAB) to reinforce the correlation between image edges and receptive fields for effective information fusion to improve the perception of detailed image features. Extensive experimental analyses affirm the competitiveness of the proposed method, showcasing its ability to balance performance and recovery effectively, even with smaller training datasets. The code is available at https://github.com/Yuii666/HADN .

In this paper, we introduce a novel system to assist 6D object pose estimation network training, which is only deployed in the training progress to optimize the network parameters, and does not work in the testing stage, called Online training and offline testing system (OTOT). OTOT consists of two modules: a feature fusion module and a supervision module. The feature fusion module fuses several feature maps from the pose estimation network in a specified order to obtain a fused feature. Then, the supervision module uses the encoder–decoder structure network to implicitly extract useful features from the fused feature and optimizes the pose estimation network online through the back-propagation mechanism. OTOT can be migrated to any network with encoder–decoder structure. The network trained with OTOT achieves 56.11% accuracy in terms of the VSD metric on the TLESS dataset using RGB inputs, compared to the 46.70% accuracy of the original network trained without OTOT. Experiments show that OTOT greatly improves the accuracy of the pose estimation network, and since OTOT is not deployed in the testing stage, it does not increase any parameters during testing and affect the original speed of the network.

Typically, deep network-based full-reference image quality assessment (FR-IQA) models compare deep features from reference and distorted images pairwise, overlooking correlations among features from the same source. We propose a dual-branch framework to capture the joint degradation effect among deep network features. The first branch uses kernel representation similarity analysis (KRSA), which compares feature self-similarity matrices via the mean absolute error (MAE). The second branch conducts pairwise comparisons via the MAE, and a training-free logarithmic summation of both branches derives the final score. Our approach contributes in three ways. First, integrating the KRSA with pairwise comparisons enhances the model's perceptual awareness. Second, our approach is adaptable to diverse network architectures. Third, our approach can guide perceptual image enhancement. Extensive experiments on 10 datasets validate our method's efficacy, demonstrating that perceptual deformation widely exists in diverse IQA scenarios and that measuring the joint degradation effect can discern appealing content deformations.

Point cloud completion 3D object detectors effectively tackle the challenge of incomplete shapes in sparse point clouds by generating pseudo points to improve detection performance. However, the absence of guidance provided by the heatmap information and the geometric shape information renders the precise recovery of object shapes an arduous task. To this end, we propose a Geometry-guided Point Generation for 3D Object Detection, named GgPG. Specifically, we first design a 3D heatmap auxiliary supervision subnetwork to enhance the quality of object proposals by capturing the actual size and position of the object within the 3D heatmap representation. Moreover, we introduce a density-aware point generation module that employs Kernel Density Estimation (KDE) to embed the point density into the grid point's feature representation, thereby enabling the completion of more precise object shapes. Our GgPG achieves progressive performance in both Waymo and KITTI benchmarks, notably GgPG outperforms PGRCNN by +1.02$\%$, +1.18$\%$, and +0.56$\%$on the vehicle, pedestrian, and cyclist under LEVEL$\_$2 mAPH classes on Waymo Open Dataset, respectively.

Text-to-image person retrieval (TPR) focuses on finding a specific person based on the textual description, and most methods implicitly assume the training image-text pairs are correctly aligned. In practice, the image-text pairs exist under-correlated or false-correlated due to the low quality of the images and annotation errors. Meanwhile, remarkable similarities between different person identities may lead to a mismatch between text and image. To tackle the two issues, we present a Visual-Language Noise Modeling (ViLNM) method that successfully captures robust cross-modal associations even with noise. Specifically, we design a Noise Token Aware (NTA) module that eliminates the words in the textual description that do not match the image, utilizing the matched words to establish a more reliable association. Besides, to enhance the recognition ability of the model for different person identities, we propose a Joint Inter and Intra-Modal Contrastive Loss (JII) and Local Aggregation (LA) module to increase the feature differences between different person identities. We conduct comprehensive experiments on three public benchmarks, and ViLNM performs best.

Low-light images often suffer from varying degrees of visual degradation. Current methods for recovering image texture details fail to rely on the self-adaptive correlation texture direction of the image itself, which leads the network to be unable to address the local texture characteristics of different images. To address this challenge, we propose a semantic-aware detail adaptive network (SDANet) that fully considers the image detail information. The network divides low-light images into high-frequency and low-frequency parts. Learning different forms of noise through a novel total variation regularization module with adaptive weights ensures that the final high-frequency part adequately integrates the texture information of the image. Simultaneously, a detail-adaptive module is incorporated to restore finer details in the resulting image. SDANet not only effectively suppresses noise in real low-light images while considering texture details but also effectively addresses the degradation of visible information, and it performs better than other state-of-the-art methods. The code is available athttps://github.com/cheer79/SDANet.

Fine-grained visual classification (FGVC) is a challenging task characterized by interclass similarity and intraclass diversity and has broad application prospects. Recently, several methods have adopted the vision Transformer (ViT) in FGVC tasks since the data specificity of the multihead self-attention (MSA) mechanism in ViT is beneficial for extracting discriminative feature representations. However, these works focus on integrating feature dependencies at a high level, which leads to the model being easily disturbed by low-level background information. To address this issue, we propose a fine-grained attention-locating vision Transformer (FAL-ViT) and an attention selection module (ASM). First, FAL-ViT contains a two-stage framework to identify crucial regions effectively within images and enhance features by strategically reusing parameters. Second, the ASM accurately locates important target regions via the natural scores of the MSA, extracting finer low-level features to offer more comprehensive information through position mapping. Extensive experiments on public datasets demonstrate that FAL-ViT outperforms the other methods in terms of performance, confirming the effectiveness of our proposed methods. The source code is available athttps://github.com/Yueting-Huang/FAL-ViT.

In the field of no-reference image quality assessment (NR-IQA), the visual masking effect has long been a challenging issue. Although existing methods attempt to alleviate the interference caused by masking by generating pseudoreference images, the quality of these images is often constrained by the accuracy and reconstruction capabilities of image restoration algorithms. This can introduce additional biases, thereby affecting the reliability of the evaluation results. To address this problem, we propose a novel generative “noise” estimation framework (GNE-Vim) that eliminates the need for pseudoreference images. Instead, it deeply decouples the distortion components from degraded images and performs quality-aware modelling of these components. During the training phase, the model leverages both reference images and distortion components to guide the learning of the true distortion distribution. In the inference phase, quality prediction is conducted directly on the basis of the decoupled distortion components, making the evaluation results more aligned with human subjective perception. The experimental results demonstrate that the proposed method achieves strong performance across datasets containing various types of distortions. The source code is publicly available at the following website: https://github.com/opencodelxt/GNE-Vim.

Detecting small objects in remote sensing images poses significant challenges to the field of computer vision, primarily stemming from the complexity of backgrounds, limitations in pixel resolution, and information loss during the feature fusion process. While general object detection has significantly advanced in recent years, remote sensing small object detection remains an unsolved problem, with existing frameworks struggling to achieve high performance at small scales. In this article, we propose a novel framework called the boundary-aware feature fusion network (BAFNet), which significantly enhances the model’s ability to represent and locate small objects precisely within complex remote sensing scenarios. First, a dual-stream attention fusion module captures complementary foreground and background cues through bidirectional context modeling. Jointly attending to objects and their surroundings enhances discriminative power for distinguishing small objects. Additionally, we incorporate a boundary-aware branch to better preserve crucial detailed information vital for small-scale objects. This auxiliary component supervises the fusion of contextual semantics and spatial information, aiding in retaining critical boundary details that are prone to loss during cross-layer feature fusion. We conducted experiments on the challenging AI-TOD, VisDrone, DIOR, and LEVIR-Ship datasets. The results demonstrate the superiority of our approach over other state-of-the-art (SOTA) object detection methods, particularly in terms of precisely identifying small objects within remote sensing images. The code is available athttps://github.com/ooo1128/BAFNet.

Vibration isolators are essential for sensitive and precise industrial applications working on instrumentation and control systems since they isolate microvibration, prevent error propagation, and enhance processing/image quality. This article proposes a novel gravity compensator (GC) for vibration isolation in satellite applications based on permanent magnets. Owing to the new magnetic circuit topology, the proposed system features a lower natural frequency and higher force density, enabling superior low-frequency isolation performance and bearing capacity compared with traditional isolators. Owing to its low natural frequency, the proposed system is superior to conventional GCs in isolating low-frequency disturbances. Moreover, the high force density enables the GC to exploit a heavier payload with less magnetic material, which is economical and space-saving. In addition, well-designed electromagnets are utilized to adjust the static levitation force, which can improve the ability to satisfy the requirements of different loads without drastically changing the stiffness of the entire system. The results suggest that the proposed GC could be utilized for satellite applications with heavy loads and limited mounting space.

Multispectral object detection has shown great promise in security and industrial applications. RGB images offer rich texture but are limited by lighting, whereas IR images excel in low light but lack texture. Current methods face challenges in accurately capturing information differences and achieving effective feature fusion across modalities. To address these issues, we propose a graph aggregation alignment network (GAANet) for multispectral object detection. GAANet consists of two key modules: the graph interaction fusion module (GIFM) and the information alignment module (IAM). GIFM uses graph representation learning to effectively process single-modality features, and the direct connection information flow mechanism guides and references low-level multimodal features, ensuring the global and comprehensive fusion of node information in the graph space. The results are then refined through the IAM for secondary calibration and alignment of corresponding local regions, ensuring accurate fusion. We also introduce an information reconstruction path (IRP) and reconstruction loss to prevent the loss of single-modality information due to multiple IAM calculations. GAANet achieves excellent fusion detection capability and significantly reduces the number of parameters, reducing the model size by 61.2% compared with that of representative baselines such as CALNet. GAANet achieves state-of-the-art results on the DroneVehicle, LLVIP, and FLIR datasets, with superior object detection accuracy. It also performs well on the unaligned DVTOD dataset, effectively capturing feature offsets across modalities through global graph perception.

Infrared small target detection has been extensively studied due to its wide range of applications. Most studies treat infrared small target detection as an independent task, either as a detection-based or a segmentation-based, failing to fully leverage the supervisory information from different annotation forms. To address this issue, we propose a multi-task mutual learning network (MTMLNet) specifically designed for infrared small targets, aiming to enhance both detection and segmentation performance by effectively utilizing various forms of supervisory information. Specifically, we design a multi-stage feature aggregation (MFA) module capable of capturing features with varying gradients and receptive fields simultaneously. Additionally, a hybrid pooling down-sampling (HPDown) module is proposed to mitigate information loss during the down-sampling process of infrared small targets. Finally, the hierarchical feature fusion (HFF) module is designed to adaptively select and fuse features from different semantic layers, learning the optimal way to fuse features across semantic layers. The results on IRSTD-1k and SIRST-V2 datasets show that our proposed MTMLNet achieves state-of-the-art (SOTA) performance in both detection-based and segmentation-based methods. The codes are available at https://github.com/YangBo0411/MTMLNet.

Vehicle trajectory prediction is gaining significant attention from academia and industry because of its vital role in autonomous driving. However, current theories face two challenges. First, they generally underperform when faced with longer historical trajectory inputs, especially in large-scale scenarios. Second, they usually ignore the temporal continuity of the target vehicle itself. To address these issues, our study proposed a novel extended temporal sequence fusion attention (ETSFA) network. This network can fully capture the information from the historical trajectory and the dynamic influences of adjacent agents. In addition, the novel dual-channel decoupled model can precisely characterize the intricate spatiotemporal interplay of on-road vehicles. Specifically, the proposed network consists of two main parts. For temporal analysis, the linear inference network (LIN) is reparameterized into complex diagonal forms at the state–space model (SSM) layer to express the linear recurrence capability, thus effectively mining the long-term historical trajectory temporal features of the target vehicle. For spatial analysis, an advanced spatial perception module (SPM) based on graph attention networks (GATs) is proposed to aggregate vehicle and intervehicle interaction features. In addition, a spatial inference module (SIM) based on a convolutional linear inference unit (CONVLIN) is customized for spatiotemporal graph features. Finally, the proposed ETSFA is trained and validated across diverse public datasets, including HighD and NGSIM, demonstrating a marked improvement in the prediction accuracy of the proposed ETSFA over existing methods.

Multispectral object detection, which combines RGB visible light and thermal infrared spectral information, has broad applications in complex environments and varying illumination conditions. However, existing methods face challenges in processing multispectral data, such as inconspicuous object features in spectral images and significant discrepancies between input modality spaces and output detection spaces. To address these issues, we propose an innovative multispectral object detection method that combines contrastive learning and a new cross-modal feature fusion module. We introduce a mask feature contrastive loss that maximizes the similarity between the box-level mask features and modal features while suppressing background responses, enabling effective representative alignment between the input and output spaces. Additionally, we propose a mask-guided attention fusion module that uses a predicted pseudo mask to guide the fusion of different modal features, enhancing object responses and reducing background noise interference. Our extensive experiments on several challenging multispectral datasets demonstrate that our proposed COFNet achieves state-of-the-art performance.

Aiming at the situation where the measurement matrix B has a flexible block decomposition, this article designs two novel distributed continuous- and discrete-time projection neurodynamic approaches to solve the basis pursuit (BP) problem for sparse recovery. These approaches only require information from each flexible block of the measurement matrix B, rather than from each row, column, or the entire matrix. First, with the aid of the primal-dual dynamical approach, projection operator, and second-order multiagent consensus condition, a novel distributed projection neurodynamic approach in continuous time (DPNA-CT-B) is proposed, and its optimality and global asymptotic stability are rigorously proved. Moreover, based on the forward and backward Euler methods and variable substitution methods, a corresponding distributed projection neurodynamic approach in discrete time (DPNA-DT-B) is designed. Finally, through sparse signal and image reconstruction experiments, the effectiveness and superiority of the proposed neurodynamic approaches are verified.

Fully connected (FC) layers play a significant role in deep neural networks (DNNs) models. Owing to the complexity of its parameters, an FC layer has sufficient capacity to manage high-dimensional tasks, so a large amount of memory and powerful computing capabilities become essential requirements. However, the large number of parameters in an FC layer greatly limits the practical application of this model. To address this problem, we apply matrix optimization to an FC layer. First, an added penalty term properly maintains the sparsity of the imposed weights. Second, a rank constraint is applied to the two components of the factorized weight matrix. Our compression algorithm can effectively reduce the number of required network parameters, which not only reduces the computational complexity of the network but also results in better generalizability on a test dataset. Finally, the effectiveness of the proposed method is verified in two different computer vision task domains. Experiments show that our sparse reduced-rank method achieves a better compression ratio with a lower accuracy loss relative to the competing approaches. The code is available at https://github.com/cheer79/Compress_FC .

The rapid development of computer vision technology for detecting anomalies in industrial products has received unprecedented attention. In this article, we propose a dual teacher–student-based discrimination (DTSD) model for anomaly detection, which combines the advantages of both embedding-based and reconstruction-based methods. First, the DTSD builds a dual teacher‒student architecture consisting of a pretrained teacher encoder with frozen parameters, a student encoder, and a student decoder. By distillation of knowledge from the teacher encoder, the two teacher‒student modules acquire the ability to capture both local and global anomaly patterns. Second, to address the issue of poor reconstruction quality faced by previous reconstruction-based approaches in some challenging cases, the model employs a feature bank that stores encoded features of normal samples. By incorporating template features from the feature bank, the student decoder receives explicit guidance to enhance the quality of reconstruction. Finally, a segmentation network is utilized to adaptively integrate multiscale anomaly information from the two teacher–student modules, thereby improving segmentation accuracy. Extensive experiments demonstrate that our method outperforms existing state-of-the-art approaches. The code of DTSD is publicly available at https://github.com/Math-Computer/DTSD .

Low-light image enhancement techniques have significantly progressed, but unstable image quality recovery and unsatisfactory visual perception are still significant challenges. To solve these problems, we propose a novel and robust low-light image enhancement method via CLIP-Fourier guided wavelet diffusion, abbreviated as CFWD. Specifically, the CFWD leverages multimodal visual-language information in the frequency domain space created by multiple wavelet transforms to guide the enhancement process. Multiscale supervision across different modalities facilitates the alignment of image features with semantic features during the wavelet diffusion process, effectively bridging the gap between the degraded and normal domains. Moreover, to further promote the effective recovery of the image details, we combine the Fourier transform based on the wavelet transform and construct a hybrid high-frequency perception module (HFPM) with a significant perception of the detailed features. This module avoids the diversity confusion of the wavelet diffusion process by guiding the fine-grained structure recovery of the enhancement results to achieve favourable metrics and perceptually oriented enhancement. Extensive quantitative and qualitative experiments on publicly available real-world benchmarks show that our approach outperforms existing state-of-the-art methods, achieving significant progress in image quality and noise suppression. The project code is available at https://github.com/hejh8/CFWD .

In this article, we propose an end-to-end Transformer-based and structure-aware dual-stream network for low-light image enhancement. First, we divide the dual-stream network into a main stream and a structure stream. The main stream is used to recover an enhanced image and supply structural information to the structure stream, whereas the structure stream is designed to extract structural features from the main stream to provide rich structural information for the enhancement process. Second, we devise a structure-gated Transformer to balance the extraction of global and local features through parallel multihead self-attention with convolution operations following a multilayer perceptron, thus extracting sufficient structural information from the main stream in the encoder part of the dual-stream network. Finally, we develop a cross-attention-based feature fusion module that divides different window sizes into distinct feature fusion stages to achieve multistage, multiscale and multitype feature fusion in the decoder part of the dual-stream network. The experimental results show that our method not only works effectively but also offers favorable computational and storage overheads.

Graphs offer a flexible structure for vision tasks, with CNNs and Transformers conditioned as two specific cases of graph structures. In CNNs, the input images are treated as graphs where only neighboring patches are connected, whereas Transformers view images as fully connected graphs. To leverage the potential of graphs in video representation learning, effective graph generation and training methods are crucial. To this end, we propose VideoGNN, which represents the video as a discrete time dynamic graph and learns the dynamic graph efficiently. Given the multitude of frames in videos, we introduce an efficient graph generation module characterized by low complexity and high quality, facilitating the transformation of videos into dynamic graphs. Additionally, we introduce a dual-view graph neural network to capture spatial and temporal information from the generated dynamic graphs. Then, a sequential model is applied to capture the long-term temporal information and generate the final frame embeddings. Experiments demonstrate that VideoGNN can achieve competitive results in terms of graph quality assessment and video downstream tasks. The codes are available at https://github.com/Dodo-D-Caster/VideoGNN .

Image semantic segmentation is the pixel-level understanding and classification of visual regions of interest. In this paper, a novel cardiac magnetic resonance image semantic segmentation architecture is presented based on a joint dual-stream convolutional neural network (CNN) called JDSCNN. The method effectively integrates multi-level, multi-layer and multi-scale context and detailed information to perform semantic segmentation. First, a semantic stream and a detail stream are used in the proposed network to extract the context and edge features respectively. Specifically, for the semantic stream, a gated fully fusion module is integrated as the skip connection to extract multi-level features. Besides, a feature pyramid module is proposed to concatenate multi-layer features in the decoder path. A feature aggregation module is also used to obtain multi-scale context. For the detail stream, a gated shape CNN exploiting multi-level context is proposed to extract edge features of the objects. Second, we integrate an inter-stream fusion module to fuse and propagate the features in each stream bidirectionally. Finally, a multi-task loss function combining the edge information and segmentation information is applied. JDSCNN is evaluated on the public cardiac image dataset, which achieves significant results for the left ventricle, right ventricle and myocardium compared with the state-of-the-art algorithm. Our code is available at: https://github.com/csathu/JDSCNN.

Versatile video coding (VVC) aims to achieve high compression but also issues like varying content/network conditions. Existing rate control (RC) methods struggle to achieve optimal quality under these complex scenarios. This paper proposes a novel RC scheme for VVC based on a linear model. The Lagrange minimization multiplier is introduced under bit budget constraints, allowing optimized bit allocation. RC optimization is formulated as a convex solution, and is derived into the optimal quantization parameter (QP) for RC. Experimental analysis demonstrates the proposed linear model-based RC algorithm performances are better compared to other state-of-the-art methods due to their use of a linear model and optimal QP determination.

With the widespread adoption of virtual reality and 360-degree video, there is a pressing need for objective metrics to assess quality in this immersive panoramic format reliably. However, existing image quality assessment models developed for traditional fixed-viewpoint content do not fully consider the specific perceptual issues involved in 360-degree viewing. This paper proposes a 360-degree image full-reference quality assessment (FR-IQA) methodology based on a multi-channel architecture. The proposed 360-degree FR-IQA method further optimizes and identifies the distorted image quality using two easily obtained useful saliency and depth-aware image features. The convolutional neural network (CNN) is designed for training. Furthermore, the proposed method accounts for predicting user viewing behaviors within 360-degree images, which will further benefit the multi-channel CNN architecture and enable the weighted average pooling of the predicted FR-IQA scores. The performance is evaluated on publicly available databases to demonstrate the advantages brought by the proposed multi-channel model in performance evaluation and cross-database evaluation experiments, where it outperforms other state-of-the-art ones. Moreover, an ablation study exhibits good generalization ability and robustness.

In this paper, a parallel decoder and a word group prediction module are proposed to speed up decoding and improve the effect of captions. The features of the image extracted by the encoder are linearly projected to different word groups, and then a unique relaxed mask matrix is designed to improve the decoding speed and the caption effect. First, since image captioning is composed of many words, sentences can also be broken down into word groups or words according to their syntactic structure, and we achieve this function through constituency parsing. Second, we make full use of the extracted features to predict the size of word groups. Then, a new embedding representing the information of the word is proposed based on word embedding. Finally, with the help of word groups, we design a mask matrix to modify the decoding process so that each step of the model can produce one or more words in parallel. Experiments on public datasets demonstrate that our method can reduce the time complexity while maintaining competitive performance.

Public computer education systems provide students essential opportunities to enhance computer literacy and information skills. However, the widespread adoption of online education technology exposes the field to several critical security risks. Threats, such as malware infections, data breaches, and other network intrusions, are all challenging the security of education systems, posing potential hazards to students’ personal information and even the entire teaching environment. To spur further work into specialized anomaly detection techniques for computer education, this paper presents an anomaly detection framework tailored for network services in computer education environments to safeguard these systems. Specifically, the proposed approach learns from large-scale online educational traffic data to classify the security state into five alert levels, enabling more granular anomaly detection and analysis. To assess their detection performance, deep learning and traditional machine learning algorithms are implemented and compared for multi-class intrusion classification. The results show that the proposed framework provides an effective security solution to bolster the integrity and stability of computer education systems against evolving network threats, enhancing threat intelligence to inform proactive security by detecting and characterizing anomalies through multilevel classification.

Deep learning-based video coding methods have demonstrated superior performance compared to classical video coding standards in recent years. The vast majority of the existing deep video coding (DVC) networks are based on convolutional neural networks (CNNs), and their main drawback is that since CNNs are affected by the size of the receptive field, they cannot effectively handle long-range dependencies and local detail recovery. Therefore, how to better capture and process the overall structure as well as local texture information in the video coding task is the core issue. Notably, the transformer employs a self-attention mechanism that captures dependencies between any two positions in the input sequence without being constrained by distance limitations. This is an effective solution to the problem described above. In this paper, we propose end-to-end transformer-based adaptive video coding (TAVC). First, we compress the motion vector and residuals through a compression network built on the vision transformer (ViT) and design the motion compensation network based on ViT. Second, based on the requirement of video coding to adapt to different resolution inputs, we introduce a position encoding generator (PEG) as adaptive position encoding (APE) to maintain its translation invariance across different resolution video coding tasks. The experiment shows that for multiscale structural similarity index measurement (MS-SSIM) metrics, this method exhibits significant performance gaps compared to conventional engineering codecs, such as [Formula: see text], [Formula: see text], and VTM-15.2. We also achieved a good performance improvement compared to the CNN-based DVC methods. In the case of peak signal-to-noise ratio (PSNR) evaluation metrics, TAVC also achieves good performance.

With the development of artificial intelligence, computer vision technology has been widely used in the fields of security monitoring, automatic driving and wisdom city. However, there has not been a research on the detection of the meridians in human bodies by using the computer vision technology. In order to promote the use of the computer vision technology in human meridian detection, this paper first releases a dataset based on human meridians, which makes up for the gap in the field of human meridian detection using image processing technology. Moreover, the human meridian detection dataset is manually annotated and proofread by experienced Traditional Chinese Medicine (TCM) practitioners according to the position and direction of the human meridians, so that the annotated human meridians are as accurate as possible. The released human meridian dataset label’s 12 meridians, including spleen meridian, pericardium meridian, stomach meridian, lung meridian, heart meridian, kidney meridian, gallbladder meridian, liver meridian, triple energizer meridian, bladder meridian, large intestine meridian and small intestine meridian. A total of 296 acupoints were labeled. At last, this paper proposes a method for data augmentation, especially for datasets with a small amount of data, wherein the data amount can be augmented by enhancing the underlying edge visual features of the data. Experimental results show that human meridians can be detected by using image processing technology, and the proposed method for data augmentation can effectively improve the detection accuracy of human meridians. The dataset can be downloaded from https://www.zksylf.com/col.jsp?id=127 .

Recent deep learning-based watermarking methods have achieved impressive results. However, they struggle with unknown distortions and often suffer from poor generalization, slow convergence, unstable training, and degraded visual quality in watermarked images. To address the above problems, this paper proposes a two-stage separable adversarial distortion (TSAD)-based robust watermarking algorithm for diffusion tensor imaging (DTI). The algorithm uses a noise-free end-to-end network in the first stage for learning and training DTI images. In the second stage, it fixes the watermark embedding network trained in the first stage, interacts the noise distortion network with the watermark extraction network to perform adversarial training for improving robustness. Experimental results show that our method achieves comparable or better robustness to seen distortions and better robustness to unseen distortions, along with enhanced stability, faster convergence, and improved visual quality in watermarked DTI images.

Hashing retrieval is a widely used technique in high spatial resolution remote sensing (RS) images due to its efficient retrieval speed and low memory overhead. However, existing hashing retrieval methods primarily focus on matching multilabel RS images, neglecting the extensive fine-grained semantic information in cross-modal RS data. Moreover, RS images exhibit notable object size differences and contain redundant features that lack effective multiscale feature extraction methods. To address these issues, we propose a novel deep multiscale fine-grained hashing (DMFH) method for cross-modal hashing retrieval of RS data. The DMFH method comprises two modules: the feature extraction module and hashing retrieval module. In the feature extraction module, we introduce a multiscale feature representation method to extract both low-level and high-level features from RS images while using a redundant optimizer to remove duplicate features. In addition, we used embedding vectors to extract fine-grained semantic information from description texts. The hashing retrieval module uses contrastive loss and triplet loss to guide the hash function toward learning and generating hash codes from extracted features. Our proposed DMFH method achieves state-of-the-art performance in two public RS image–text datasets (RSICD and RSITMD) through extensive experiments and ablation studies.

Video coding is a video compression technique that compresses the original video sequence to produce a smaller archive file or reduce the transmission bandwidth under constraints on the visual quality loss. Rate control (RC) plays a critical role in video coding. It can achieve stable stream output in practical applications, especially real-time video applications such as video conferencing or game live streaming. Most RC algorithms either directly or indirectly characterise the relationship between the bit rate (R) and quantisation (Q) and then allocate bits to every coding unit so as to guarantee the global bit rate and video quality level. This paper comprehensively reviews the classic RC technologies used in international video standards of past generations, analyses the mathematical models and implementation mechanisms of various schemes, and compares the performance of recent state-of-the-art RC algorithms. Finally, we discuss future directions and new application areas for RC methods. We hope that this review can help support the development, implementation, and application of RC for new video coding standards.

Single-frame infrared small target detection is considered to be a challenging task, due to the extreme imbalance between target and background, bounding box regression is extremely sensitive to infrared small target, and target information is easy to lose in the high-level semantic layer. In this article, we propose an enhancing feature learning network (EFLNet) to address these problems. First, we notice that there is an extremely imbalance between the target and the background in the infrared image, which makes the model pay more attention to the background features rather than target features. To address this problem, we propose a new adaptive threshold focal loss (ATFL) function that decouples the target and the background, and utilizes the adaptive mechanism to adjust the loss weight to force the model to allocate more attention to target features. Second, we introduce the normalized Gaussian Wasserstein distance (NWD) to alleviate the difficulty of convergence caused by the extreme sensitivity of the bounding box regression to infrared small target. Finally, we incorporate a dynamic head mechanism into the network to enable adaptive learning of the relative importance of each semantic layer. Experimental results demonstrate our method can achieve better performance in the detection performance of infrared small target compared to the state-of-the-art (SOTA) deep-learning-based methods. The source codes and bounding box annotated datasets are available athttps://github.com/YangBo0411/infrared-small-target.

Object detection plays a key role in helping to enable industrial quality control and safety monitoring. This article introduces a lightweight and efficient transformer-based object detection network called the encoder-free DEtection TRansformer (EF-DETR). This novel architecture enhances the DETR model through a redesigned network structure, leading to improved accuracy in object detection and a more lightweight network. To address the issue of suboptimal object detection accuracy, especially for small objects in the DETR model, we introduce a multiscale feature extractor and a high-efficiency feature fusion module. These components facilitate the direct extraction of fine-grained features, thereby enabling effective object detection. Departing from the use of a high-complexity encoder structure, we explore the utilization of an encoder-free neck structure to reduce the network's computational complexity. In addition, to expedite convergence, denoising training is incorporated into the decoder. This article presents extensive experiments, and the EF-DETR demonstrates strong performance on the MS COCO2017 dataset compared to other popular models.

Cardiac image segmentation is essential when applying biomedical informatics to improve industrial healthcare applications. To extract context and detailed information more efficiently and further improve cardiac image segmentation accuracy, we present a novel deep dual-stream convolutional neural network (CNN) for cardiac image semantic segmentation in this article. We use a body stream and a shape stream, respectively, in this method. First, in the body stream we propose integrating a gated fully fusion module to fuse multilevel features in the encoder and decoder paths. In addition, we integrate a feature aggregation module to extract the multiscale context. Second, in the shape stream, we propose using a gated shape CNN exploiting multilevel context to extract detailed information, such as boundary and shape features. Finally, we apply a multitask loss function to align the predicted masks with the ground truth labels. Our experiments on the public cardiac magnetic resonance image dataset show significant performance in the left and right ventricular cavities and myocardium compared to the state-of-the-art algorithms.

Optimizing traffic light configuration is viewed as a method to increase the traffic throughput in urban cities. Recent studies have employed reinforcement learning to optimize the traffic light configuration. However, the assumption of these studies is oversimplified as all traffic lights are controlled by one unified policy. In the real world, the situation becomes more complicated as a city may deploy more than one traffic light policy due to the different development stages of the city. In this work, we propose a novel multiagent reinforcement learning method, called MixLight, which aims to learn the traffic light configuration under an environment of mixed policies. Our contribution is twofold. First, we propose an executor-guide dual network, in which the guide network changes the executor network optimization direction via reward shaping. Second, we improve the centralized training and decentralized execution framework for the traffic light environment, which reduces the exploration space of agents and decreases the nonstationary during training process. This assists the agents in achieving a cooperative strategy based on their local observations during the execution. Experiments on real-world and synthetic datasets verify the superiority of our proposed method.

This study aims to develop advanced and training-free full-reference image quality assessment (FR-IQA) models based on deep neural networks. Specifically, we investigate measures that allow us to perceptually compare deep network features and reveal their underlying factors. We find that distribution measures enjoy advanced perceptual awareness and test the Wasserstein distance (WSD), Jensen-Shannon divergence (JSD), and symmetric Kullback-Leibler divergence (SKLD) measures when comparing deep features acquired from various pretrained deep networks, including the Visual Geometry Group (VGG) network, SqueezeNet, MobileNet, and EfficientNet. The proposed FR-IQA models exhibit superior alignment with subjective human evaluations across diverse image quality assessment (IQA) datasets without training, demonstrating the advanced perceptual relevance of distribution measures when comparing deep network features. Additionally, we explore the applicability of deep distribution measures in image super-resolution enhancement tasks, highlighting their potential for guiding perceptual enhancements. The code is available on website. (https://github.com/Buka-Xing/Deep-network-based-distribution-measures-for-full-reference-image-quality-assessment).

In this paper, we propose a graph-represented image distribution similarity (GRIDS) index for full-reference (FR) image quality assessment (IQA), which can measure the perceptual distance between distorted and reference images by assessing the disparities between their distribution patterns under a graph-based representation. First, we transform the input image into a graph-based representation, which is proven to be a versatile and effective choice for capturing visual perception features. This is achieved through the automatic generation of a vision graph from the given image content, leading to holistic perceptual associations for irregular image regions. Second, to reflect the perceived image distribution, we decompose the undirected graph into cliques and then calculate the product of the potential functions for the cliques to obtain the joint probability distribution of the undirected graph. Finally, we compare the distances between the graph feature distributions of the distorted and reference images at different stages; thus, we combine the distortion distribution measurements derived from different graph model depths to determine the perceived quality of the distorted images. The empirical results obtained from an extensive array of experiments underscore the competitive nature of our proposed method, which achieves performance on par with that of the state-of-the-art methods, demonstrating its exceptional predictive accuracy and ability to maintain consistent and monotonic behaviour in image quality prediction tasks. The source code is publicly available at the following website https://github.com/Land5cape/GRIDS.

Aircraft detection holds significant importance in both civilian and military fields, such as air transportation control and battlefield situational awareness. Deep learning-based methods for aircraft detection can obtain promising detection performance with sufficient and labeled samples. However, current public aircraft datasets are mainly from top-down perspective images, lacking multi-perspectives samples, which limits their applicability. Therefore, we construct a multi-perspectives aircraft dataset (MAD), encompassing 10 distinct classes, and making up of 13,205 images and 18,908 instances. Moreover, the imbalanced and similar classes are common issues in aircraft detection. We design an adaptive threshold focal loss (ATFL) function to alleviate the class imbalance through dynamically adjusting the loss weights between different classes. Meanwhile, the dynamic visual center (DVC) module which can effectively capture both local and global information of target is proposed to distinguish the aircraft classes that share similar feature. Finally, we evaluated several state-of-the-art methods on the constructed MAD dataset as baselines for future research.The experimental results verify that the recognition performance of the current state-of-the-art model can be improved after the application of ATFL and DVC. The dataset and code are available athttps://github.com/YangBo0411/aircraft-detection.

Images are composed of “things” (i.e., structured objects) and “stuff” (i.e., textured surfaces), which have completely different effects on the human visual system (HVS). A good image quality assessment (IQA) method should fully consider the visual salience effects of image structures and the masking effects of image textures. In this article, we propose a perceptual quality analysis model using structure separation and high-order moments (SSHMPQA) in the deep domain. First, we use a total variation (TV) model to separate the perceptual structures in images from their deep feature maps, thereby maintaining meaningful object shapes with texture suppression and defining perceptual structure-aware distances in the deep domain. Then, we use the first- to fourth-order moments to calculate the mean, skewness and kurtosis of the probability distributions of the deep features. On this basis, we define a perceptual texture-aware distance in the deep domain. We then formulate the final model by solving a well-defined perceptual optimization problem. The proposed SSHMPQA model has good interpretability and is data-driven; moreover, the model does not require a complex and long training process because the optimization problem is convex and has an exact analytical solution. To verify the effectiveness of our model, comprehensive experiments are conducted. The experimental results show that the proposed model is superior to other state-of-the-art traditional and deep learning-based full-reference (FR) IQA methods.

Enhancing low-light image visibility is a critical task in computer vision since it helps to improve input for high-level algorithms. High-quality images typically have clear structural information. In previous studies, due to the lack of proper structural guidance, restored images had some problems, such as unclear structural areas and overexposed or underexposed local areas. To address the above problems, in this paper, we introduce a coefficient of variation (COV) with excellent performance in maintaining structural information, and then we propose a low-light image enhancement method that utilizes the COV to extract structural information from images. First, we apply a traditional retinex model to estimate both reflectance and illumination. Second, we use the COV to indicate the degree of dispersion of the input sample, which enables us to obtain a robust structure-distinguishing weight map for low-light images. The weight map is adaptively divided to obtain a structural weight map, which is then used to enhance the gradient image. This process is applied before the reflectance layer of the retinex model. Finally, the result is obtained by using the block coordinate descent method. According to extensive experiments, outstanding results can be achieved by our proposed method in terms of both subjective and objective evaluation metrics in comparison with other state-of-the-art methods. The source code is available at our website.

The proliferation of multimodal big data in cities provides unprecedented opportunities for modeling and forecasting urban problems, such as crime prediction and house price prediction, through data-driven approaches. A fundamental and critical issue in modeling and forecasting urban problems lies in identifying suitable spatial analysis units, also known as city region partition. Existing works rely on subjective domain knowledge for static partitions, which is general and universal for all tasks. In fact, different tasks may need different city region partitions. To address this issue, we propose JLPR , a task-oriented framework for J oint L earning of region P artition and R epresentation. To make partitions fit tasks, JLPR integrates the region partition into the representation model training and learns region partitions using the supervision signal from the downstream task. We evaluate the framework on two prediction tasks (i.e., crime prediction and housing price prediction) in Chicago. Experiments show that JLPR consistently outperforms state-of-the-art partitioning methods in both tasks, which achieves above 25% and 70% performance improvements in terms of mean absolute error for crime prediction and house price prediction tasks, respectively. Additionally, we meticulously undertake three visualization case studies, which yield profound and illuminating findings from diverse perspectives, demonstrating the remarkable effectiveness and superiority of our approach.

Foggy days limit the functionality of outdoor surveillance systems. However, it is still a challenge for existing methods to maintain the uniformity of defogging between image regions with a similar depth of field and large differences in appearance. To address above problem, this article proposes a regional gradient constrained prior (RGCP) for defogging that uses the piecewise smoothing characteristic of the scene structure to achieve accurate estimation and reliable constraint of the transmission. RGCP first derives that when adjacent similar pixels in the fog image are aggregated and spatially divided into regions, clusters of region pixels in RGB space conform to a chi-square distribution. The offset of the confidence boundary of the clusters can be regarded as the initial transmission of each region. RGCP further uses a gradient distribution to distinguish different regional appearances and formulate an interregional constraint function to constrain the overestimation of the transmission in the flat region, thereby maintaining the consistency between the estimated transmission map and the depth map. The experimental results demonstrate that the proposed method can achieve natural defogging performance in terms of various foggy conditions.

Images captured under low-light conditions are characterized by lower visual quality and perception levels than images obtained in better lighting scenarios. Studies focused on low-light enhancement techniques seek to address this dilemma. However, simple image brightening results in significant noise, blurring, and color distortion. In this paper, we present a low-light enhancement (LLE) solution that effectively synergizes Retinex theory with deep learning. Specifically, we construct an efficient image gradient map estimation module based on convolutional networks that can efficiently generate noise-free image gradient maps to assist with denoising. Second, to improve upon the traditional optimization model, we design a matrix-preserving optimization method (MPOM) coupled with deep learning modules, and it exhibits high speed and low memory consumption. Third, we incorporate image structure, image texture, and implicit prior information to optimize the enhancement process for low-light conditions and overcome prevailing limitations, such as oversmoothing, significant noise, and so forth. Through extensive experiments, we show that our approach has notable advantages over the existing methods and demonstrate superiority and effectiveness, surpassing the state-of-the-art methods by an average of 1.23 dB in PSNR for the LOL and VE-LOL datasets. The code for the proposed method is available in a public repository for open-source use: https://github.com/luxunL/DRNet .

RGBT tracking is gaining popularity due to its ability to provide effective tracking results in a variety of weather conditions. However, feature specificity and complementarity have not been fully used in existing models that directly fuse the correlation filtering response, which leads to poor tracker performance. In this article, we propose correlation filters with adaptive modality weight and cross-modality learning (AWCM) ability to solve multimodality tracking tasks. First, we use weighted activation to fuse thermal infrared and visible modalities, and the fusion modality is used as an auxiliary modality to suppress noise and increase the learning ability of shared modal features. Second, we design modal weights through average peak-to-correlation energy coefficients to improve model reliability. Third, we propose consistency in using the fusion modality as an intermediate variable for joint learning consistency, thereby increasing tracker robustness via interactive cross-modal learning. Finally, we use the alternating direction method of multipliers algorithm to produce a closed solution and conduct extensive experiments on the RGBT234, VOT-TIR2019, and GTOT tracking benchmark datasets to demonstrate the superior performance of the proposed AWCM against compared to existing tracking algorithms. The code developed in this study is available at the following website. 1

This article investigates a fully distributed inertial neurodynamic approach for sparse recovery. The approach is based on proximal operators and inertia items. It aims to solve the$L_{1}$-norm minimization problem with consensus and linear observation constraints over directed communication networks. The proposed neurodynamic approach has the advantages of only requiring the communication network to be directed and weight-balanced, does not involve a central processing node and global parameters, which means that no single node can access the entire network and observe it at any time, so it is fully distributed. To effectively deal with the nonsmooth objective function,$L_{1}$-norm, the proximal operator method is used here. For efficiently handling linear observation and consensus constraints, a primal-dual method is applied to the inertial dynamic system. With the aid of maximal monotone operator theory and Baillon-Haddad lemmas, it reveals that the trajectories of our approach can converge to consensus solution at the optimal solution, provided that the distributed parameters satisfy technical conditions. In addition, we aim to demonstrate the weak convergence of the trajectories in our proposed neurodynamic approach toward the zeros of the optimal operator in Hilbert space, using Opial’s lemma. Finally, comparative experiments on sparse signal and image recovery confirm the efficiency and effectiveness of our proposed neurodynamic approach.

This paper explores how to schedule energy to optimize the information freshness in wireless powered communication networks (WPCNs) when considering channel interference among adjacent sensor nodes. We introduce Age of Information (AoI) to quantitatively evaluate the information freshness and formulate the AoI optimization problem. Unlike prior works focusing on system optimization for WPCNs while ignoring channel interference in energy transfer, this work reveals situations where channel interference among adjacent sensor nodes cannot be neglected and explores optimizing information freshness with interference consideration. To take the phenomena into account, we propose an energy scheduling solution to detect the channel interference and then judiciously determine the energy and time allocation for individual sensor nodes to improve the AoI performance as well as the system throughput. We implement a multi-node WPCN testbed to validate the functional correctness of the proposed solution, and extensive experiments have demonstrated the effectiveness of the proposed solution. The experimental results show that the proposed solution can reduce the average AoI by 54.7% and the average throughput by 49.8% on average compared to the state-of-the-art solutions.

Cross-modality person re-identification (cm-ReID) aims to match pedestrian images from visible and infrared cameras. Most existing methods ignore data bias due to different cameras and views and overlook the strong dependence between feature maps that hinders modal alignment. In this paper, we propose a unified method named Joint Robust Representation and Generalization Enhancement (RRGE) to alleviate the above issues. First, we propose a robust representation module (RRM), which can improve the model’s robustness for the global context, camera, and view change perturbations. Second, we propose a generalization enhancement module (GEM), which uses channel-level dropout to alleviate the dependencies between feature maps to improve the model’s generalization. Moreover, we balance the number of different modalities in each batch. Our method outperforms other state-of-the-art methods in terms of cross-modality person re-identification tasks.

In this paper, we propose a lightweight multi-scale based attention network (MBAN) for single-image super-resolution (SISR). First, a deep feature transform block (DFTB) is designed for multi-scale feature extraction; this block combines group convolution and improved channel attention (ICA) for performance purposes while remaining sufficiently lightweight. Second, a dual multi-scale attention block (DMAB) is proposed for long-range information interaction; this block employs different window sizes for self-attention (SA) and short connections between different branches to achieve multiscale attention interaction. Finally, our MBAN is constructed by cascaded multi-scale based attention blocks (MBABs) that perform detail restoration; these blocks simultaneously extract multi-scale local features and integrate multi-scale global features with the DFTBs and DMABs. Extensive experiments suggest the superiority of our MBAN over the state-of-the-art (SOTA) lightweight SR methods in terms of both quantitative metrics and visual quality.

In recent years, single image super-resolution (SISR) methods using convolutional neural networks (CNN) have achieved satisfactory performance. Nevertheless, the large model scale and the slow inference speed of these methods greatly limit the application scenarios. In this paper, we propose a two-stage three-dimensional attention network (ATTNet) for lightweight image super-resolution. First, we put forward the spatial feature encoder–decoder (SFE-D) with a spatial attention mechanism. Next, the channel transposed attention module (CTAM) with a channel self-attention mechanism is designed. Both the modules are used for fine feature extraction in the low-resolution stage. Finally, the content-based pixel recombination module (CPRM) is proposed to reconstruct the detailed content with a joint attention mechanism in the high-resolution stage. According to our experimental results, significant performance in terms of the quantitative metrics and the subjective visual quality can be achieved on average compared with the state-of-the-art lightweight SISR algorithms.

In image enhancement, maintaining the texture and attenuating noise are worth discussing. To address these problems, we propose a low-light image enhancement method with contrast increase and illumination smooth. First, we calculate the maximum map and the minimum map of RGB channels, and then we set maximum map as the initial value for illumination and introduce minimum map to smooth illumination. Second, we use the histogram-equalized version of the input image to construct the weight for the illumination map. Third, we propose an optimization problem to obtain the smooth illumination and refined reflectance. Experimental results show that our method can achieve better performance compared to the state-of-the-art methods.

Automated object measurement is becoming increasingly important due to its ability to reduce manual costs, increase production efficiency, and minimize errors in various fields. In this paper, we present a novel approach to three-dimensional (3D) object measurement based on point cloud modeling. Our method introduces a fast point cloud modeling computation framework consisting of five stages: coordinate centralization, rotation and translation, noise filtering, plane projection, and geometric computation. Furthermore, we propose a fast convex hull optimization algorithm to reduce the high complexity problem of traditional convex hull calculation. Our extensive experiments demonstrate that our approach outperforms existing methods in terms of measurement error rate and time savings, with a maximum time saving of 31.03% under certain error conditions.

We propose a full-reference image quality assessment (FR-IQA) method by incorporating low-level and high-level image features. First, in contrast to the preexisting deep IQA methods, which only use the features extracted by the deep network, we not only use the image gradient to replace the low-level features in the first two stages of the deep network, but also combine them with the middle-stage features of the deep network to construct the new low-level features. The deep features of shallow layers contain some unwanted noise and further result in a decline in IQA performance. Second, we combine the global features extracted by the self-attention-based model with the semantic features extracted by the convolutional neural network to form the high-level features. Instead of directly using the self-attention-based model trained on the classification task, we first train a no-reference (NR) IQA regression model on a larger dataset and then use the global features of this NR-IQA model. The self-attention-based model can capture the internal connections of the image and is more effective in extracting global information due to its larger perceptual field. In the final pooling stage, we combine the average pooling and the standard deviation pooling to obtain the dispersion and concentration of the similarity maps for a more comprehensive description of quality. Experiments show that our FR-IQA method is able to obtain competitive results on three standard IQA datasets.

Community search over bipartite graphs has attracted significant interest recently. In many applications such as the user–item bipartite graph in e-commerce and customer–movie bipartite graph in movie rating website, nodes tend to have attributes. However, the previous community search algorithms on bipartite graphs ignore attributes, thus making them to return results with poor cohesion with respect to their node attributes. In this paper, we study the community search problem on attributed bipartite graphs. Given a query vertex [Formula: see text], we aim to find the attributed [Formula: see text]-communities of [Formula: see text], where the structure cohesiveness of the community is described by the [Formula: see text]-core model, and the attribute similarity of two groups of nodes in the subgraph is maximized. In order to retrieve attributed communities from bipartite graphs, we first propose a basic algorithm composed of two steps: the generation and verification of candidate keyword sets, and then two improved query algorithms Inc and Dec are proposed. Inc is proposed considering the anti-monotonicity property of attributed bipartite graphs, then we adopt different generating methods and verify the order of candidate keyword sets and propose the Dec algorithm. After evaluating our solutions on eight large graphs, the experimental results demonstrate that our methods are effective and efficient in querying the attributed communities on bipartite graphs.

Session-based recommendation (SBR) emphasizes mining user interests to predict the next click based on recent interactions within sessions. Most current SBR methods suffer from insufficient interactive information problems and fail to distinguish session representations with high similarities, which can neglect the inherent features within sessions. To fill the gap, we propose a triplet mining enhanced graph neural networks (TME-GNN) approach to enhance the recommendation systems by mining structural and inherent information. Technically, we first generate anchor, positive and negative embeddings based on the given session and set a triplet mining task to improve the recommendation task with subtle features by pushing positive pairs close and pulling negative pairs away. Second, to robust the model, we employ a self-supervised auxiliary task by adding dynamic perturbations to the embedding space. We conduct extensive experiments to demonstrate the superiority of our method against other state-of-the-art algorithms. Our implementations are available on the following site https://github.com/Info4Rec/TME-GNN .

This paper investigates two accelerated primal-dual mirror dynamical approaches for smooth and nonsmooth convex optimization problems with affine and closed, convex set constraints. In the smooth case, an accelerated primal-dual mirror dynamical approach (APDMD) based on accelerated mirror descent and primal-dual framework is proposed and accelerated convergence properties of primal-dual gap, feasibility measure and the objective function value along with trajectories of APDMD are derived by the Lyapunov analysis method. Then, we extend APDMD into two distributed dynamical approaches to deal with two types of distributed smooth optimization problems, i.e., distributed constrained consensus problem (DCCP) and distributed extended monotropic optimization (DEMO) with accelerated convergence guarantees. Moreover, in the nonsmooth case, we propose a smoothing accelerated primal-dual mirror dynamical approach (SAPDMD) with the help of smoothing approximation technique and the above APDMD. We further also prove that primal-dual gap, objective function value and feasibility measure along with trajectories of SAPDMD have the same accelerated convergence properties as APDMD by choosing the appropriate smooth approximation parameters. Later, we propose two smoothing accelerated distributed dynamical approaches to deal with nonsmooth DEMO and DCCP to obtain accelerated and efficient solutions. Finally, numerical and comparative experiments are given to demonstrate the effectiveness and superiority of the proposed accelerated mirror dynamical approaches.

Fast and accurate image retrieval is an important and challenging task in massive image data scenarios. As the core technology of image retrieval tasks, deep metric learning aims at learning effective embedding representations that possess two properties among data points: positive concentrated and negative separated. In this work, we propose a multilevel similarity-aware method based on deep local descriptors for deep metric learning. We take the rich interclass similarity relationship based on the deep local invariant descriptors from the data into account to optimize sampling strategies for mining informative samples. The method dynamically adjusts the margin between data points to better match the true similarity relationship between classes. Specifically, for images in a batch, we first obtain deep local descriptors and calculate the similarity matrix of the channel, pixel, and spatial levels. Then, depending on the calculated comprehensive similarity matrix, we propose a multilevel similarity-aware loss function through the deviation between pairwise distance and violate margin to make full use of informative samples. The experimental results demonstrate that our proposed method outperforms other state-of-the-art methods in terms of fine-grained image retrieval and clustering tasks.

Advanced 360$^\circ$video streaming is essential for high-quality communication and industrial video applications, supporting novel and interactive visual experiences that promote consumption. However, due to the inherent fluctuations in communication networks, hardware resources, and power costs, a tradeoff needs to be achieved between occupying network bandwidth and guaranteeing 360$^\circ$visual quality. To address this issue, this article proposes a quality-aware global optimization solution that combines different types of sensory characteristics to further enhance the 360$^\circ$streaming efficiency in cellular networks. First, considering the characteristics of 360$^\circ$video, each frame is divided into three different regions based on a proposed field-of-view prediction method. Second, a new region-based rate-distortion model reflecting the bitrate-quality relationship is proposed. The divided regions are represented by different rate-distortion models. In addition, a rate-distortion model parameter update strategy with robustness to region changes is proposed to further guarantee transmission performance. Finally, we propose a globally optimized adaptive bitrate allocation algorithm to optimize 360$^\circ$mobile streaming, which uses both the rate-distortion models and viewpoint prediction results. Evaluation results indicate that the proposed method outperforms state-of-the-art approaches in terms of several quality of experience objectives under various network conditions.

In this paper, a joint decision tree and visual feature optimization rate control scheme for ultrahigh-definition (UHD) versatile video coding (VVC) is proposed. First, we design a new rate-distortion (R-D) model for UHD videos, and we establish a decision-tree-based multiclass classification scheme to improve the prediction accuracy of the R-D model by fully considering visual features. Second, based on the proposed R-D model, the globally optimal solution is obtained through convex optimization. Finally, we embed our algorithm into the latest VVC reference software, VTM 10.2. According to our experimental results, compared with the latest algorithm in VTM 10.2 and other state-of-the-art algorithms, our method can achieve significant bit rate reductions while maintaining a given peak signal-to-noise ratio (PSNR) or structural similarity index measure (SSIM).

Rapid processing of abandoned objects is one of the most important tasks in road maintenance. Abandoned object detection heavily relies on traditional object detection approaches at a fixed location. However, detection accuracy and range are still far from satisfactory. This study proposes an abandoned object detection approach based on vehicular ad-hoc networks (VANETs) and edge artificial intelligence (AI) in road scenes. We propose a vehicular detection architecture for abandoned objects to achieve task-based AI technology for large-scale road maintenance in mobile computing circumstances. To improve detection accuracy and reduce repeated detection rates in mobile computing, we propose a detection algorithm that combines a deep learning network and a deduplication module for high-frequency detection. Finally, we propose a location estimation approach for abandoned objects based on the World Geodetic System 1984 (WGS84) coordinate system and an affine projection model to accurately compute the positions of abandoned objects. Experimental results show that our proposed algorithm achieves an average accuracy of 99.57% and 53.11% on the two datasets, respectively. Additionally, our whole system achieves real-time detection and high-precision localization performance on real roads.

This article proposes a frequency-based structure and texture decomposition model in a Retinex-based framework for low-light image enhancement and noise suppression. First, we utilize the total variation-based noise estimation to decompose the observed image into low-frequency and high-frequency components. Second, we use a Gaussian kernel for noise suppression in the high-frequency layer. Third, we propose a frequency-based structure and texture decomposition method to achieve low-light enhancement. We extract texture and structure priors by using the high-frequency layer and a low-frequency layer, respectively. We present an optimization problem and solve it with the augmented Lagrange multiplier to generate a balance between structure and texture in the reflectance map. Our experimental results reveal that the proposed method can achieve superior performance in naturalness preservation and detail retention compared with state-of-the-art algorithms for low-light image enhancement. Our code is available on the following website. 1

This paper proposes a global optimization solution that combines different types of perceptual information to further improve transmission efficiency. First, a new rate-distortion (R-D) model is proposed to reflect the characteristics of 360degree video. Second, a globally optimized adaptive bitrate control algorithm is proposed using both the R-D models to adjust the bitrate for each tile in a segment. Finally, a new model parameter updating strategy that is robust to quality variations is proposed to further reduce prediction errors. Comparison results indicate that the proposed method can outperform state-of-the-art methods in terms of various quality of experience (QoE) objectives1.

User interest modeling in sequential recommendation is to learn the underlying interests from the user's historical interaction records, and then predict a list of items that the user might interact with. Existing studies treat each item as an integral unit, and mainly focus on extracting useful information from the item sequence directly. However, they neglect the fact that a user usually interacts with an item because of some attributes of it, and the cumulative result of attributes reflects the user's more specific interests. In this work, we propose a novel approach named Attribute-driven Interest Modeling for sequential recommendation (AIM), and explore user latent interests in a fine-grained manner. More specifically, our attribute-driven module captures more accurate interests from the attribute sequence, where each attribute first aggregates information from its neighbor items with a graph convolutional network. Then, the learned multiple interests are aggregated with the base interest extracted from the original item sequence to form user representation. Finally, the user representation is used to calculate the ranking score and make predictions. Extensive experiments on three real-world datasets show that our model outperforms the state-of-the-art baseline by 0.49%$\sim$8.61%.

Existing deep learning-based full-reference IQA (FR-IQA) models usually predict the image quality in a deterministic way by explicitly comparing the features, gauging how severely distorted an image is by how far the corresponding feature lies from the space of the reference images. Herein, we look at this problem from a different viewpoint and propose to model the quality degradation in perceptual space from a statistical distribution perspective. As such, the quality is measured based upon the Wasserstein distance in the deep feature domain. More specifically, the 1D Wasserstein distance at each stage of the pre-trained VGG network is measured, based on which the final quality score is performed. The deep Wasserstein distance (DeepWSD) performed on features from neural networks enjoys better interpretability of the quality contamination caused by various types of distortions and presents an advanced quality prediction capability. Extensive experiments and theoretical analysis show the superiority of the proposed DeepWSD in terms of both quality prediction and optimization. The implementation of our method is publicly available at https://github.com/Buka-Xing/DeepWSD.

Object detection at the edge side is a common task in various environments. The deployment of convolutional neural networks in intelligent edge systems is very challenging because of the highly constrained main-memory space. This study aims at operating neural networks with a reduced memory requirement. The basic idea is that tasks of the same type would involve the same critical subnetwork. We propose identifying the critical network connections by considering the importance of channels. During runtime, the proposed method detects the task types and timely swaps the model parameters of the critical subnetworks from the external storage into dynamic random access memory (DRAM). Compared with conventional network pruning, the proposed approach further reduced the DRAM requirement by 34.6% while maintaining a high inference accuracy.

A contrastive distortion-level learning-based no-reference image-quality assessment (NR-IQA) framework is proposed in this study to further effectively model various distortion types with the same or different distortion levels. The proposed method aims to improve the prediction accuracy of NR-IQA. The proposed method consists of three parts: multiscale distortion-level representation learning, single-image NR-IQA, and a representation affinity module, which can reduce NR-IQA computational complexity while maintaining a low-distortion representation of high-distortion inputs. The proposed NR-IQA method aims to extract distributional features of samples in real distorted images and predict ambiguity based on distortion-level learning. Experimental results show that by comparing on many NR-IQA data sets the proposed method can outperform state-of-the-art methods.

Visual retrieval has been a significant technology in the computer vision task. Visual feature descriptors are the key to the visual retrieval. The famous local feature descriptor is called the Scale Invariant Feature Transform (SIFT), which can keep invariant mapping for the scale, rotate and simulate images. To utilize effectively the SIFT feature descriptor for visual matching on different hardware platforms, this paper proposes an accelerated SIFT algorithm based on the SIFT feature computing principle of the general multi-core platform. First, our multi-core task allocation method introduces the WFM theory into task assignment for each core to improve the core computing resource utilization for high-efficient parallel computing. Then, to improve the efficiency of picture matching, we introduce global geometric constraints condition to optimal picture matching for the multi-core parallelization approach. Experimental results show that the proposed approach can save on average 87.31% on the Intel X86 platform, compared to the single-core time. Also, our approach can save on average 33.79% on the Raspberry Pi platform, compared to the single-core time.

In this paper, we propose a multi-modal sparse tracking by jointing timing and modal consistency to locate the target location with the similarity of multiple local appearances. First, we propose an alignable patching strategy for red-green-blue (RGB) color mode and thermal infrared mode to adapt to the local changes of the target. Second, we propose a consistency expression of the corresponding aligned patches between the modes and the correlation of the gaussian mapping within mode to reconstruct the target judgment likelihood function. Finally, we propose an updating scenario based on timing correlation and mode sparsity to fit with the target changes. According to the experimental results, significant improvement in terms of tracking accuracy can be achieved on average compared with the state-of-the-art algorithms. The source code of our algorithm is available on https://github.com/Liincq/tracker .

Local search algorithms are widely applied in solving large-scale distributed constraint optimization problem (DCOP). Distributed stochastic algorithm (DSA) is a typical local search algorithm to solve DCOP. However, DSA has some drawbacks including easily falling into local optima and the unfairness of assignment choice. This paper presents a novel local search algorithm named VLSs to solve the issues. In VLSs, sampling according to the probability corresponding to assignment is introduced to enable each agent to choose other promising values. Besides, each agent alternately performs a greedy choice among multiple parallel solutions to reduce the chance of falling into local optima and a variance adjustment mechanism to guide the search into a relatively good initial solution in a periodic manner. We give the proof of variance adjustment mechanism rationality and theoretical explanation of impact of greed among multiple parallel solutions. The experimental results show the superiority of VLSs over state-of-the-art DCOP algorithms.

Collecting users’ historical data such as movie watching and music listening, and mining frequent items from them, can improve the utility of smart services, but there is also a risk of compromising user privacy. Local differential privacy is a strict definition of privacy and has been widely used in various privacy-preserving data collection scenarios. However, the accuracy of existing locally differentially private frequent items mining methods decreases significantly with the increase in the dimensions of data to be collected. In this paper, we propose a new locally differentially private frequent item mining method for high-dimensional data, which decreases the dimension used for data perturbation by grouping the contents and improving the interference matrix generation method, so as to improve the data reconstruction accuracy. The experimental results show that our proposed method can significantly improve the accuracy of frequent item mining and provide a better trade-off between privacy and accuracy compared with existing methods.

In this paper, we propose a structure-preserving and denoising low-light enhancement method that uses the coefficient of variation. First, we use the coefficient of variation to process the original low-light image, which is used to obtain the enhanced illumination gradient reference map. Second, we use the total variation (TV) norm to regularize the reflectance gradient, which is used to maintain the smoothness of the image and eliminate the artifacts in the reflectance estimation. Finally, we combine the above two constraint terms with the Retinex theory, which contains the denoising regular term. The final enhanced and denoised low-light image is obtained by iterative solution. Experimental results show that our method can achieve superior performance in both subjective and objective assessments compared with other state-of-the-art methods (the source code is available at: https://github.com/bbxavi/SPDLEM .).

Three-dimensional (3D) human pose estimation aims to locate 3D keypoints of individuals from given input RGB images. For two-dimensional (2D) human pose estimation problems, majority methods inferring 2D poses are from 2D heatmaps. However, it is hard to extend this method to 3D poses inferring area which makes computational loads increase sharply. To address the above problem, we propose STM-CNN method to estimate reconstruction coefficient matrix to calculate the final 3D pose instead of estimating 3D heatmaps to decrease the computational loads. First, STM-CNN does a preprocessing procedure to calculate a set of shape and weight bases. Second, STM-CNN infers a 2D matrix called reconstruction coefficient from the STM-CNN architecture. Third, STM-CNN utilizes the preprocessing shape and weight bases and estimated reconstruction coefficient matrix to calculate the final 3D pose. Meanwhile, STM-CNN method achieves better performances compared with the state-of-the-art methods on Human3.6M.

3D human pose estimation describes estimating 3D articulation structure of a person from an image or a video. The technology has massive potential because it can enable tracking people and analyzing motion in real time. Recently, much research has been conducted to optimize human pose estimation, but few works have focused on reviewing 3D human pose estimation. In this paper, we offer a comprehensive survey of the state-of-the-art methods for 3D human pose estimation, referred to as pose estimation solutions, implementations on images or videos that contain different numbers of people and advanced 3D human pose estimation techniques. Furthermore, different kinds of algorithms are further subdivided into sub-categories and compared in light of different methodologies. To the best of our knowledge, this is the first such comprehensive survey of the recent progress of 3D human pose estimation and will hopefully facilitate the completion, refinement and applications of 3D human pose estimation.

Power consumption attacks monitoring on artificial intelligence (AI) chips play a critical role in the vehicular AI systems. However, most of the current monitoring and management methods focus on the trustworthiness of industrial equipment instead of resource-constrained edge devices. To address the above problem, a closed-loop module for monitoring and management of vehicular AI chips based on fitting and filtering to resist power consumption attacks is proposed in this paper. First, considering the characteristics of power, we propose a raw data correction approach for power monitoring to monitor abnormal power consumption. Second, we address the challenging problem of precision temperature monitoring to monitor the abnormal temperature of the chip, especially in a wide temperature range. Finally, the established method is applied to attack surveillance and transformed into a power consumption management problem solved by dynamic voltage and frequency scaling (DVFS) technology. As the experimental results reveal, compared with existing methods of power and temperature monitoring and power consumption control in wide temperature, our method can achieve significantly improved monitoring and managing performance.

Train operation control in urban railways is challenging due to its high dynamics, complex environment, and level of comfort and safety. To address these challenges, in this article, the authors propose a new deep reinforcement-based train operation (DRTO) method which includes: 1) A deterministic deep reinforcement learning algorithm, 2) a dynamic incentive system, which is used to ensure safe operation in a multitrain environment, and 3) an event-driven method, which is used to improve the DRTO performance based on an event-driven strategy. To evaluate the performance, we thoroughly compare the proposed method with other operation control solutions on both synthetic and real datasets. Our results demonstrate that DRTO is effective in: 1) Decreasing the energy consumption of train operation, 2) increasing passenger comfort, and 3) achieving a good tradeoff between efficiency and safety. In addition, the effectiveness of the event-driven strategy and the dynamic incentive system is demonstrated in the experiments.

ResNetsare widely used in the intrusion detection system (IDS) of software-defined industrial network to construct accurate intelligence detection of network attacks. However, the IDS based on ResNets has a long detecting interval because of the fine-grained operator and intermediate outcomes of the multi-branch architecture of ResNets. To address this problem, in this article, we propose Juggler-ResNet with a fusible residual structure that preserves the feature extraction ability of the residual structure and enables equivalent transformation to linear topology to support low latency inference service in the industrial application (e.g., malicious network behavior detection, fault diagnosis, etc.). First, we propose a fusible multibranch residual structure to avoid gradient vanishing problems in the training phase. Second, we convert it to linear-topology by using a set of equivalent fusion operators. Finally, the linear-topology model is deployed to accelerate inference speed. Our experimental results on CIFAR-10 and CIFAR-100 show that fusible residual structure can achieve 2.08-4.3x acceleration with state-of-the-art level accuracy performance.

A 360-degree streaming system can provide immersive, interactive, and autonomous experiences surrounding the user by means of viewpoint changes to see different angles of a 360-degree video. However, due to the limited capacity and highly dynamic conditions of cellular networks, high-resolution 360-degree video playback over mobile devices often suffers from playback freezing, and bandwidth waste is inevitably incurred in delivering out-of-view video data. In this paper, a hybrid control scheme is presented for segment-level continuous bitrate selection and tile-level bitrate allocation for 360-degree streaming over mobile devices to increase users’ quality of experience. First, a deep reinforcement learning (RL) method is proposed to predict the segment bitrate and avoid playback freezing. Second, a viewpoint-prediction-map-based cooperative bargaining game theory is proposed for bitrate allocation optimization to choose a suitable bitrate for each tile to reduce unreasonable bandwidth waste. The proposed scheme is compared with state-of-the-art approaches under a wide variety of mobile network conditions with multiple viewpoint traces and 360-degree video contents. The experimental results indicate that the proposed method outperforms the compared state-of-the-art approaches in terms of various experimental objectives on mobile devices.

A joint reinforcement learning (RL) and game theory method is presented for segment-level continuous bitrate selection and tile-level bitrate allocation in tile-based 360-degree streaming to increase users’ quality of experience (QoE). First, a viewpoint prediction method based on single-user (SU) viewpoint traces and the saliency map (SM) model is presented to model viewing behaviours. Second, an RL method is proposed to predict segment bitrate and a cooperative bargaining game theory is proposed for bitrate allocation optimization to choose a suitable bitrate for every tile with the help of the viewpoint prediction map. Performance evaluation results indicate that the proposed method can outperform the state-of-the-art methods in terms of different QoE objectives.

In this paper, we propose a seamless multimodal binary learning method for cross-modal retrieval. First, we utilize adversarial learning to learn modality-independent representations of different modalities. Second, we formulate loss function through the Bayesian approach, which aims to jointly maximize correlations of modality-independent representations and learn the common quantizer codebooks for both modalities. Based on the common quantizer codebooks, our method performs efficient and effective cross-modal retrieval with fast distance table lookup. Extensive experiments on three cross-modal datasets demonstrate that our method outperforms state-of-the-art methods. The source code is available at https://github.com/zhouyu1996/DAQN.

In this paper, we propose distribution-aware hierarchical weighting (DHW) method for deep metric learning. First, we formulate the distributions of different classes according to the form of gaussian curves, and update distributions as the training process. Second, depending on the learnable distribution, we propose a loss function named distribution-aware loss with dynamic mining margins and hierarchical degrees of weights to make full use of samples. The experimental results show that our algorithm outperforms other state-of-the-art methods in terms of retrieval and clustering tasks. Code is available at https://github.com/zhuyinong1/DHW-master.

In this paper, a quality of experience (QoE)-based neural live streaming method with dynamic adaptive video quality control is developed to improve streaming performance. First, the dynamic adaptive streaming issue is formulated as a Markov decision process (MDP) problem. Second, an reinforcement learning (RL)-based approach is proposed as an appropriate solution, where the client functions as an RL agent and the environment is made up of various networks. User QoE is the reward by mutual consideration of video quality and play-back state. Finally, to optimize the total reward, the RL algorithm chooses the required video quality for each video segment. Experimental results show that the proposed RL-based streaming algorithm outperforms state-of-the-art schemes in terms of both temporal and visual QoE metrics by a noticeable margin while guaranteeing application-level fairness when multiple clients share a bottlenecked network. The code is available on the following website: https://github.com/OpenCode007/ICME2021.

Liver lesion segmentation from abdomen computed tomography (CT) with deep neural networks remains challenging due to the small volume and the unclear boundary. To effectively tackle these problems, in this paper, we propose a cascaded deeply supervised convolutional networks (CDS-Net). The cascaded deep supervision (CDS) mechanism uses auxiliary losses to construct a cascaded segmentation method in a single network, focusing the network attention on pixels that are more difficult to classify, so that the network can segment the lesion more effectively. CDS mechanism can be easily integrated into standard CNN models and it helps to increase the model sensitivity and prediction accuracy. Based on CDS mechanism, we propose a cascaded deep supervised ResUNet, which is an end-to-end liver lesion segmentation network. We conduct experiments on LiTS and 3DIRCADb dataset. Our method has achieved competitive results compared with other state-of-the-art ones.

Perceptual video coding (PVC) optimization has been an important video coding technique, which can be consistent with the perception characteristics of the human visual system (HVS). Currently, PVC schemes incorporating the just noticeable distortion (JND) model can obtain better performance gain in all PVC schemes. To further accelerate the JND computation for real-time video coding applications (e.g. surveillance video coding and conference video coding), this paper proposes a fast perceptual surveillance video coding (PSVC) scheme based on background model-driven JND estimation method. First, to utilize the surveillance scene characteristics, the computation complexity of JND estimation can be significantly decreased by reusing the content complexity of background regions. Then we apply the perceptive video coding scheme into the background modeling-based surveillance video codec. The proposed scheme adopts background modeling frame as background anchor. Experimental results show that the proposed scheme can yield remarkable time saving of 42.33% maximum and on average 34.76% with approximate bitrate reductions and similar subjective quality, compared to HEVC and other state-of-the-art schemes.

With rapid development of intelligent video surveillance systems based on cloud computing devices and edge computing devices in Cyber-Physical-Social Systems, massive surveillance video data has brings enormous challenge for video storage and transmission. However, existing surveillance video coding approaches hardly utilize intelligent video analysis results for improving video coding. This paper proposed a surveillance video coding scheme for traffic scene based on vehicle knowledge and shared library by cloud-edge computing in Cyber-Physical-Social Systems. Firstly, in order to provide the object library for synchronous application at the encode and decode side offline, a generation method of shared long-term foreground reference object library is proposed by using the existing large-scale monitoring vehicle object datasets. Then, to meet the requirement of low complexity and high-performance coding, a virtual foreground reference picture generation method with coding-oriented object retrieval is proposed. Experimental results show that the proposed scheme can obtain the satisfactory effect of the virtual foreground reference picture. Also, it can yield remarkable bit rate reductions, compared to HEVC.

Cross-modal retrieval aims to retrieve the related items from various modalities with respect to a query from any type. The key challenge of cross-modal retrieval is to learn more discriminative representations between different category, as well as expand to an unseen class retrieval in the open world retrieval task. To tackle the above problem, in this paper, we propose a prototype learning-based discriminative feature learning (PLDFL) to learn more discriminative representations in a common space. First, we utilize a prototype learning algorithm to cluster these samples labeled with the same semantic class, by jointly taking into consideration the intra-class compactness and inter-class sparsity without discriminative treatments. Second, we use the weight-sharing strategy to model the correlations of cross-modal samples to narrow down the modality gap. Finally, we apply the prototype to achieve class-incremental learning to prove the robustness of our proposed approach. According to our experimental results, significant retrieval performance in terms of mAP can be achieved on average compared to several state-of-the-art approaches.

Rate control (RC) plays a critical role in the transmission of high-quality video data under certain bandwidth restrictions in High Efficiency Video Coding (HEVC). Most current HEVC RC algorithms based on spatio-temporal information for rate-distortion (R-D) model parameters cannot effectively handle the cases with dynamic video sequences that contain fast moving objects, significant object occlusion or scene changes. In this paper, we propose an RC method based on deep reinforcement learning (DRL) for dynamic video sequences in HEVC to improve the coding efficiency. First, the rate control problem is formulated as a Markov decision process (MDP) problem. Second, with the MDP model, we develop a DRL-based algorithm to find the optimal quantization parameters (QPs) by training a deep neural network. The resulting intelligent agent selects the optimal RC strategy to reduce distortion, buffer and quality fluctuations by observing the current state of the encoder. The asynchronous advantage actor-critic (A3C) method is used to solve the MDP problem. Finally, the proposed DRL-based RC method is implemented in the newest video coding standard. Experimental results show that the proposed method offers substantially enhanced RC accuracy and consistently outperforms HEVC reference software and other state-of-the-art algorithms.

Distantly supervised relation extraction has been widely applied in knowledge base construction due to its less requirement of human efforts.However, the automatically established training datasets in distant supervision contain low-quality instances with noisy words and overlapped relations, introducing great challenges to the accurate extraction of relations.To address this problem, we propose a novel Regularized Attentive Capsule Network (RA-CapNet) to better identify highly overlapped relations in each informal sentence.To discover multiple relation features in an instance, we embed multi-head attention into the capsule network as the low-level capsules, where the subtraction of two entities acts as a new form of relation query to select salient features regardless of their positions.To further discriminate overlapped relation features, we devise disagreement regularization to explicitly encourage the diversity among both multiple attention heads and low-level capsules.Extensive experiments conducted on widely used datasets show that our model achieves significant improvements in relation extraction.

With numerous coding tools adopted in the forthcoming Versatile Video Coding (VVC) standard, much less work has been dedicated to study the corresponding Rate-Distortion (R-D) characteristics. This paper proposes a new quadratic R-D model for Versatile Video Coding. In particular, based on the proposed model, a new R-λ relationship is derived and used for frame level rate control. The rate control algorithm is implemented on VTM 2.0 platform for intra coding scenarios. Compared to the default rate control algorithm in VTM 2.0, experimental results show that proposed rate control algorithm can achieve 0.77% BD-BR reduction with similar control accuracy.

In this paper, we proposed a novel hybrid active contour driven by double-weighted signed pressure force method for image segmentation. First, the Legendre polynomials and global information are integrated into the signed pressure force (SPF) function and a coefficient is applied to weight the effect degrees of the Legendre term and global term. Second, by introducing a weighted factor as the coefficient of inside and outside region fitting center, the curve can be optimally evolved to the interior and branches of the region of interest (ROI). Third, a new edge stopping function is adopted to robustly capture the edge of ROI and speed up the multi-object image segmentation. Experiments show that the proposed method can achieve better accuracy for images with noise, inhomogeneous intensity, blur edge and complex branches, in the meanwhile, it also controls the time-consuming effectively and is insensitive to the initial contour position.

In this paper, we propose a method to segment the valid region of fisheye images. First, we construct an objective function with three terms, which are the region driving term, the edge driving term and the length regularization term. Second, we minimize this objective function by a modified gradient descent method to find the best segmentation result. Our method can achieve valid region segmentation by making use of both region information and edge information. Experiments show that the proposed method can deal with blurred edges, halation noise and incomplete valid region problems.

In this paper, we propose an adaptive variable exponent level set method based on Legendre polynomials for object segmentation in complex visual environment. First, we use a set of Legendre basis functions to approximate the region intensity, which enable us to accommodate heterogeneous objects. Second, an improved function is presented to update exponent adaptively and ensures the image gradient information embedding into the model easily. The proposed method is robust to low contrast, blurred boundaries, noise and the 10-cation of initial contour, and sufficient in handling large scale intensity variations. Experimental results demonstrate that the proposed method can achieve relatively high segmentation accuracy and less computational time.

In this paper, we propose a novel fault diagnosis (FD) approach for micro-electromechanical systems (MEMS) inertial sensors that recognize the fault patterns of MEMS inertial sensors in an end-to-end manner. We use a convolutional neural network (CNN)-based data-driven method to classify the temperature-related sensor faults in unmanned aerial vehicles (UAVs). First, we formulate the FD problem for MEMS inertial sensors into a deep learning framework. Second, we design a multi-scale CNN which uses the raw data of MEMS inertial sensors as input and which outputs classification results indicating faults. Then we extract fault features in the temperature domain to solve the non-uniform sampling problem. Finally, we propose an improved adaptive learning rate optimization method which accelerates the loss convergence by using the Kalman filter (KF) to train the network efficiently with a small dataset. Our experimental results show that our method achieved high fault recognition accuracy and that our proposed adaptive learning rate method improved performance in terms of loss convergence and robustness on a small training batch.

High dynamic range (HDR) video compression technology, which is capable of delivering a wider range of luminance and a larger colour gamut than standard dynamic range (SDR) technology, has been widely used in recent years in many fields, including industrial image processing, digital entertainment, and machine vision. Rate control (RC) is of paramount importance to HDR compression and transmission; accordingly, an RC scheme for HDR in High Efficiency Video Coding (HEVC) is proposed in this paper. First, considering the HDR characteristics, we propose an HDR-Visual Difference Predictor (VDP)-2-based rate-distortion (R-D) model to improve the coding performance. Second, we directly utilize $\lambda $ rather than the bit rate in the optimization process to obtain the optimal solution. Finally, we propose a new model parameter estimation method to further reduce the RC errors. According to our experimental results, significant bit rate reductions in terms of HDR-VDP-2, the Video Quality Metric (VQM) and the mean peak-signal-to-noise ratio (mPSNR) can be achieved on average compared with the state-of-the-art algorithm used in HM16.19.

In this paper, we propose a just noticeable distortion (JND)-based perceptual rate control method for high efficiency video coding (HEVC). First, the JND factor of a coding unit has been mathematically shown to be an approximation of the average pixel-level JND weight, which means that it can also be used as a weight for bitrate allocation. Second, rate-distortion (R-D) modelling is conducted based on the JND factor. Finally, the proposed R-D model is integrated into an existing rate control framework to improve the coding efficiency, and the proposed algorithm is implemented in the newest video coding standard. As the experimental results reveal, compared with HEVC reference software, our algorithm achieves significantly improved coding performance, subjective coding quality and bitrate accuracy.

This paper proposes a novel sparsity-based error concealment (EC) algorithm which integrates the Gauss Bayes model and singular value decomposition for high efficiency video coding (HEVC). Under the sequential recovery framework, pixels in missing blocks are successively reconstructed in Gauss Bayes mode. We find that the estimation error follows the Gaussian distribution in HEVC, so the error pixel estimation problem can be transferred to a Bayesian estimation. We utilize the singular value decomposition (SVD) technique to select sample pixels, which yields high estimation accuracy and reduces estimation error. A new recovery order based on confidence is established to resolve the error propagation problem. Compared to other state-of-the-art EC algorithms, experimental results show that the proposed method gives better reconstruction performance in terms of objective and subjective evaluations. It also has significantly lower complexity.

In this paper, we propose a coding tree unit (CTU)-level rate control scheme from the perspective of SSIM-based rate-distortion optimization to improve the coding efficiency. First, we establish the SSIM-based rate-distortion model based on the divisive normalization scheme, which characterizes the relationship between the local visual quality and the coding bits. Then, the established model is applied to the CTU-level rate control and transformed into a global optimization problem solved by convex optimization. Finally, a new model parameter updating strategy for the CTU-level rate control is presented that is robust to scene variations. Our algorithm can achieve optimal CTU-level bit allocation given the bit-rate budget. The experimental results show that our algorithm substantially enhances the coding performance and consistently outperforms both the rate control scheme in the HEVC reference software and existing algorithms in terms of rate-perceptual distortion performance using different test configurations.

Rate control is a crucial consideration in high-efficiency video coding (HEVC). The estimation of model parameters is very important for coding tree unit (CTU)-level rate control, as it will significantly affect bit allocation and thus coding performance. However, the model parameters in the CTU-level rate control sometimes fails because of inadequate consideration of the correlation between model parameters and complexity characteristic. In this study, we establish a novel complexity correlation-based CTU-level rate control for HEVC. First, we formulate the model parameter estimation scheme as a multivariable estimation problem; second, based on the complexity correlation of the neighbouring CTU, an optimal direction is selected in five directions for reference CTU set selection during model parameter estimation to further improve the prediction accuracy of the complexity of the current CTU. Third, to improve their precision, the relationship between the model parameters and the complexity of the reference CTU set in the optimal direction is established by using least square method (LS), and the model parameters are solved via the estimated complexity of the current CTU. Experimental results show that the proposed algorithm can significantly improve the accuracy of the CTU-level rate control and thus the coding performance; the proposed scheme consistently outperforms HM 16.0 and other state-of-the-art algorithms in a variety of testing configurations. More specifically, up to 8.4% and on average 6.4% BD-Rate reduction is achieved compared to HM 16.0 and up to 4.7% and an average of 3.4% BD-Rate reduction is achieved compared to other algorithms, with only a slight complexity overhead.