In multimodal semantic segmentation tasks of urban street scenes, existing methods lack modeling of intermodal structural alignment and semantic cooperation between architectures, leading to insufficient fusion feature representations. To address this issue, this article proposes a novel structural optimization network: a hybrid embedding segment anything model (SAM) guided feedback network (GFNet). This network is based on the SAM framework and achieves multimodal structural alignment by transforming the semantic prior (SP) extractor through module-level fine-tuning of the image encoder. Furthermore, this article proposes a cross-architecture knowledge transfer (CAKT) mechanism, injecting the structural awareness capability of SAM into the backbone features of each layer, achieving dual optimization of alignment and enhancement. To address the issues of intermodal heterogeneity and semantic conflict, this article combines complementary fusion at different frequencies and cross-modal similarity enhancement strategies to achieve fine-grained semantic fusion and consistency modeling, supplemented by a dual-supervised constraint mechanism to improve modal independence and robustness. On several challenging datasets, mAcc is improved by about 5%, and GFNet demonstrates the superior segmentation performance and robustness compared to existing methods. Our code will be released to the public athttps://github.com/WBangG/GFNet

The crowd density estimation has recently attracted attention from various industries. For efficient computing in the real-time applications, this paper proposes a lightweight relation-aware network based on self-contrastive distillation for red‒green‒blue, thermal (RGB-T) crowd counting. In order to better deal with irregular area images taken by drones, a novel multi-relationship graph reasoning module is designed in the network. By modeling and reasoning the graph node relationship on multi-layer, deep interaction in resolution is achieved from the pixel level, and rich multimodal information is obtained. Furthermore, to make full use of the information on self-attention, the gaussian selection module is introduced, in which gaussian filtering is used to obtain valuable information and convert it into a probabilistic form for further optimized modal communication. This paper also proposes a new cyclic contrast distillation training method. By dividing positive samples and negative samples for comparative learning and training, the intermediate features are optimized to influence the final output, and the final output is then transferred to the underlying features for self-distillation. In this way, the intermediate information is strengthened and a new training cycle is formed. The feature expression of each layer is greatly optimized without increasing parameters. Finally, a large number of experiments have shown that our model performs well on the RGB-T crowd dataset, and the relevant code will be made public on https://github.com/WBangG/LRANet.

Accurate detection of surface defects on magnetic chips is a necessary and difficult task, especially for small defects, which lack discriminative and robust features due to their small size and weak characteristics. At present, the low accuracy of small defect detection seriously restricts the development of automated visual inspection and needs to be solved urgently. Thus, we propose a novel small defects detection network based on context-weighted aggregation and feature multiscale loop fusion. First, a context-weighted aggregation module (CAM) that enriches feature representations by combining context and attention mechanisms is proposed. We believe that any further improvements will be futile if robust feature representations cannot be obtained. Second, inspired by the mechanism of looking and thinking twice, we aggregate left-right feedback connections into feature pyramids and creatively propose a loop-shaped feature pyramid network (Loop-FPN), enabling multiscale features to be fused up and down and connected left and right. This loop-shaped structure makes the connection of each layer more direct and allows the features at each scale to be fully integrated, which improves the utilization of multiscale features and facilitates the detection of small defects. Finally, we apply the proposed network to practical detection and the results show that our network achieves 97.57% precision, 91.91% recall, and 98.39% AP, which are 0.13%, 1.43%, and 0.99% higher than the current best-performing comparative methods, respectively. Note to Practitioners—Current vision inspection technology has low accuracy and poor stability in small defect detection, which cannot meet the industrial inspection requirements. Small defect detection has become the biggest challenge in the field of visual inspection and has seriously restricted its development. Our proposed network can solve this problem well by mining small defect context, fusing multiscale features and utilizing attention mechanisms, and has been successfully applied in magnetic chip production line. Furthermore, we developed an image acquisition system that can capture defects on all surfaces with high accuracy and without dead space, allowing our network to not only detect small defects on magnetic chips of varying sizes and irregular shapes, but also to adapt to changes in lighting conditions, backgrounds, and viewpoints. Our work provides an effective, reliable, and convenient quality control solution for magnetic chip production.

With the success of self-supervised learning, multimodal foundation models have rapidly adapted a wide range of downstream tasks driven by vision and language (VL) pre-training. State-of-the-art methods achieve impressive performance by pre-training on large-scale datasets. However, bridging the semantic gap between the two modalities remains a non-negligible challenge for VL tasks. In this work, we propose an efficient computation framework for multimodal alignment by introducing a novel visual semantic module to further improve the performance of the VL tasks. Specifically, we propose a flexible model, namely Artificial-Spiking Hierarchical Networks (ASH-Nets), which combines the complementary advantages of Artificial Neural Network (ANN) and Spiking Neural Network (SNN) to enrich visual semantic representations. In particular, a visual concrete encoder and a semantic abstract encoder are constructed to learn continuous and discrete latent variables to enhance the flexibility of semantic encoding. Considering the spatiotemporal properties of SNN modeling, we introduce a contrastive learning method to optimize the inputs of similar samples. This can improve the computational efficiency of the hierarchical network, while the augmentation of hard samples is beneficial to the learning of visual representations. Furthermore, the Spiking to Text Uni-Alignment Learning (STUA) pre-training method is proposed, which only relies on text features to enhance the encoding ability of abstract semantics. We validate the performance on multiple well-established downstream VL tasks. Experiments show that the proposed ASH-Nets achieve competitive results. Our code is available on GitHub (https://github.com/ZSYTJ/ASH-Nets).

Goal-conditioned hierarchical reinforcement learning (HRL) presents a promising approach for enabling effective exploration in complex, long-horizon reinforcement learning (RL) tasks through temporal abstraction. Empirically, heightened interlevel communication and coordination can induce more stable and robust policy improvement in hierarchical systems. Yet, most existing goal-conditioned HRL algorithms have primarily focused on the subgoal discovery, neglecting interlevel cooperation. Here, we propose a novel goal-conditioned HRL framework named Guided Cooperation via Model-Based Rollout (GCMR; code is available at https://github.com/HaoranWang-TJ/GCMR_ACLG_official), aiming to bridge interlayer information synchronization and cooperation by exploiting forward dynamics. First, the GCMR mitigates the state-transition error within off-policy correction via model-based rollout, thereby enhancing sample efficiency. Second, to prevent disruption by the unseen subgoals and states, lower level Q-function gradients are constrained using a gradient penalty with a model-inferred upper bound, leading to a more stable behavioral policy conducive to effective exploration. Third, we propose a one-step rollout-based planning, using higher level critics to guide the lower level policy. Specifically, we estimate the value of future states of the lower level policy using the higher level critic function, thereby transmitting global task information downward to avoid local pitfalls. These three critical components in GCMR are expected to facilitate interlevel cooperation significantly. Experimental results demonstrate that incorporating the proposed GCMR framework with a disentangled variant of hierarchical reinforcement learning guided by landmarks (HIGL), namely, adjacency constraint and landmark-guided planning (ACLG), yields more stable and robust policy improvement compared with various baselines and significantly outperforms previous state-of-the-art (SOTA) algorithms.

Construction waste classification (CWC) is a crucial and challenging task. However, this promising CWC system has not yet been fully implemented worldwide. The similar appearance of construction waste makes it difficult to be classified. We believe that if the characteristics of the appearance between different materials can not be fully extracted and effectively propagated, any further improvements of CWC models will be futile. To this end, this paper develops a texture and pixel gradient convolutional network (TPCNet) to capture material features. We find that the texture features as the additional input features can provide intrinsic fine-grained patterns. First, we propose the early-stage fusion strategy, where natural texture features and RGB are concatenated together as input. Second, we propose a texture and pixel gradient convolution (TPC) module to consider subtle texture information. TPC focuses on the texture guidance and introduces it to the modern CNNs, which enjoy the best of both worlds. We extend TPC to GTPC using a lossless synthetic large kernel (SLK) to achieve global contextual dependencies and seamlessly incorporate channel-wise adaptability. Moreover, we propose a two-stage feature pyramid network (TFPN) to fuse richer and denser spatial multi-scale contextual information. We also construct a large-scale real-world dataset and a production line to demonstrate the effectiveness of our model. Note to Practitioners—This paper is motivated by the problem in real-world construction waste classification task. The construction waste have similar 2D appearance makes them hard to be classified. Current models often blindly feed raw RGB images into CNNs instead of considering specific features, such as textured material surface textures. This paper proposes the texture guidance mechanism and introduces it to the training of CNN. Furthermore, we propose a lossless synthetic large kernel (SLK) to achieve global texture information. By introducing texture information as a basis for classification, our model can effectively classify similarly shaped construction waste and improve segmentation performance, thus providing the robot with more accurate location information.

RGB-D semantic segmentation can be advanced with convolutional neural networks due to the availability of Depth data. Although objects cannot be easily discriminated by just the 2D appearance, with the local pixel difference and geometric patterns in Depth, they can be well separated in some cases. Considering the fixed grid kernel structure, CNNs are limited to lack the ability to capture detailed, fine-grained information and thus cannot achieve accurate pixel-level semantic segmentation. To solve this problem in the CNN structure, we propose a Pixel Difference Convolutional Network (PDCNet) to capture detailed intrinsic patterns by aggregating both intensity and gradient information in the local range for Depth data and global range for RGB data, respectively. Precisely, PDCNet consists of a Depth branch and an RGB branch. For the Depth branch, we propose a Pixel Difference Convolution (PDC) to consider local and detailed geometric information in Depth data via aggregating both intensity and gradient information. For the RGB branch, we contribute a lightweight Cascade Large Kernel (CLK) to extend PDC, namely CPDC, to enjoy global contexts for RGB data and further boost performance. Consequently, the local and global pixel differences from both modal data are seamlessly incorporated into PDCNet during the information propagation process. Experiments on three challenging benchmark datasets,$i.e.$, NYUDv2 (78.4 Pixel Acc., 53.5 mIoU), SUN RGB-D (83.3 Pixel Acc., 49.6 mIoU) and SID Dataset (83.1 Pixel Acc., 61.4 mIoU) reveal that our PDCNet achieves state-of-the-art performance for the semantic segmentation task.

Visual question answering (VQA) has been intensively studied as a multimodal task, requiring efforts to bridge vision and language for correct answer inference. Recent attempts have developed various attention-based modules for solving VQA tasks. However, the performance of model inference is largely bottlenecked by visual semantic comprehension. Most existing detection methods rely on bounding boxes, remaining a serious challenge for VQA models to comprehend and correctly infer the causal nexus of contextual object semantics in images. To this end, we propose a finer model framework without bounding boxes in this work, termedLooking Out of Instance Semantics (LOIS)to address this crucial issue. LOIS can achieve more fine-grained feature descriptions to generate visual facts. Furthermore, to overcome the label ambiguity caused by instance masks, two types of relation attention modules: 1) intra-modality and 2) inter-modality, are devised to infer the correct answers from different visual features. Specifically, we implement a mutual relation attention module to model sophisticated and deeper visual semantic relations between instance objects and background information. In addition, our proposed attention model can further analyze salient image regions by focusing on important word-related questions. Experimental results on four benchmark VQA datasets prove that our proposed method has favorable performance in improving visual reasoning capability.

Deepfake poses a serious threat to the reliability of judicial evidence and intellectual property protection. Existing detection methods either blindly utilize deep learning or use biosignal features, but neither considers spatial and temporal relevance of face features. These methods are increasingly unable to resist the growing realism of fake videos and lack generalization. In this paper, we identify a reliable fingerprint through the consistency of AR coefficients and extend the original PPG signal to 3-dimensional fingerprints to effectively detect fake content. Using these reliable fingerprints, we propose a novel model-agnostic method to expose Deepfake by analyzing temporal and spatial faint synthetic signals hidden in portrait videos. Specifically, our method extracts two types of faint information, i.e., PPG features and AR features, which are used as the basis for forensics in temporal and spatial domains, respectively. PPG allows remote estimation of the heart rate in face videos, and irregular heart rate fluctuations expose traces of tampering. AR coefficients reflect pixel-wise correlation and spatial traces of smoothing caused by up-sampling in the process of generating fake faces. Furthermore, we employ two ACBlock-based DenseNets as classifiers. Our method provides state-of-the-art performance on multiple deep forgery datasets and demonstrates better generalization.

Model-based reinforcement learning (MBRL) approaches have demonstrated great potential in handling complex tasks with high sample efficiency. However, MBRL struggles with asymptotic performance compared to model-free reinforcement learning (MFRL). In this paper, we present a long-horizon policy optimization method, namely model-based deterministic policy gradient (MBDPG), for efficient exploitation of the learned dynamics model through multi-step gradient information. First, we approximate the dynamics of the environment with a parameterized linear combination of an ensemble of Gaussian distributions. Moreover, the dynamics model is equipped with a memory module and trained on a multi-step prediction task to reduce cumulative error. Second, successful experience is used to guide the policy at the early stage of training to avoid ineffective exploration. Third, a clipped double value network is expanded in the learned dynamics to reduce overestimation bias. Finally, we present a deterministic policy gradient approach in the model that backpropagates multi-step gradient along the imagined trajectories. Our method shows higher sampling efficiency than the state-of-the-art MFRL methods while maintaining better convergence performance and time efficiency compared to the SOAT MBRL.

Planning in learned dynamics models has proven to have great potential in improving sample efficiency. However, learning the latent representation that embeds to model dynamics from high-dimensional observation is still challenging. We here propose a model-based policy gradient method, that quickly learns an optimal policy directly from pixel frames. First, the dynamics model is learned with a SENet architecture, that explicitly incorporates attention and gating mechanism to differentiate features for the latent representation. Second, the policy in the early stage is guided with successful experience to extract the intention of the experts thus to speed up the convergence. Finally, we use model rollout to decrease the value estimation bias and multi-step policy gradients to update the policy. Our approach outperforms state-of-the-art algorithms on multiple benchmark tasks in sampling efficiency and convergence performance.

Because of the redundant information contained in the EEG signals, the classification accuracy of motor imagery may be greatly reduced. The channel selection method helps to remove task-independent EEG signals, thereby improving the performance of the BCI system. However, the brain regions associated with motor imagery in different subjects are not exactly same, and the method of channel selection depends on the feedback of classification results. Aiming at the above problems, this paper proposed a new method of channel selection based on backward elimination and threshold. First, we calculated the variance of channels between categories based on the covariance matrix. The backward selection and threshold selection methods were then used to retain the highly differentiated channels. We also used different criterion to evaluate the discriminability between different classes. The selected EEG signals used common spatial pattern (CSP) and support vector machine (SVM) to calculate the classification accuracy. The efficacy of the proposed approach was examined using three data sets. The proposed approach has achieved 77.82% accuracy on the BCI Competition IV data set IIa, 86.02% accuracy on the BCI Competition III data set IIIa, and 86.86% accuracy on the binary class BCI Competition III data set IVa. The performance of the proposed algorithm is compared with other existing algorithms. The results of our experiments demonstrated that the proposed algorithm produces a higher classification accuracy compared to the other algorithms using lesser number of channels.

Summary Motor imagery (MI) is an important control paradigm in the field of brain‐computer interface (BCI), which enables the recognition of personal intention. So far, numerous methods have been designed to classify EEG signal features for MI task. However, deep neural networks have been seldom applied to analyze EEG signals. In this study, two novel kinds of deep learning schemes based on convolutional neural networks (CNN) and Long Short‐Term Memory (LSTM) were proposed for MI‐classification. The frequency domain representations of EEG signals were obtained using short time Fourier transform (STFT) to train models. Classification results were compared between conventional algorithm, CNN, and LSTM models. Compared with two other methods, CNN algorithms had shown better performance. These conclusions verified that CNN method was promising for MI‐based BCIs.

Most of the traditional just-noticeable-distortion (JND) models in pixel domain compute the JND threshold by incorporating the spatial luminance adaptation effect and the textures contrast masking effect. Recently, with the rapid development of the computable models of visual attention, researchers started to improve the JND model by considering visual saliency of images, a foveated spatial JND model (FSJND) was proposed by incorporating the traditional visual characteristics and fovea characteristic of human eyes to enhance JND thresholds. However, the thresholds computed by the FSJND model may be overestimated for some high resolution images. In this paper, we proposed a new JND profile in pixel domain, in which a multi-level modulation function is built to reflect the effect of hierarchically selective visual attention on JND thresholds. The contrast masking is also considered in our modulation function to obtain more accurate JND thresholds. Compared with the lasted JND profiles, the proposed model can tolerate more distortion and has much better perceptual quality. The proposed JND model can be easily applied in many areas, such as compression, error protection, and so on.

Peer‐to‐peer (P2P) systems provide a new solution to distributed information and resource sharing because of its outstanding properties in decentralization, dynamics, flexibility, autonomy, and cooperation, summarized as DDFAC in this paper. After a detailed analysis of the current P2P literature, this paper suggests to better exploit peer social relationships and peer autonomy to achieve efficient P2P structure design. Accordingly, this paper proposes Self‐organizing peer‐to‐peer social networks (SoPPSoNs) to self‐organize distributed peers in a decentralized way, in which neuron‐like agents following extended Hebbian rules found in the brain activity represent peers to discover useful peer connections. The self‐organized networks capture social associations of peers in resource sharing, and hence are called P2P social networks. SoPPSoNs have improved search speed and success rate as peer social networks are correctly formed. This has been verified through tests on real data collected from the Gnutella system. Analysis on the Gnutella data has verified that social associations of peers in reality are directed, asymmetric and weighted, validating the design of SoPPSoN. The tests presented in this paper have also evaluated the scalability of SoPPSoN, its performance under varied initial network connectivity and the effects of different learning rules.

In this paper, an algorithm that forms a dynamic and self-organizing network is demonstrated. The hypothesis of this work is that in order to achieve a resilient and adaptive peer-to-peer (P2P) network, each network node must proactively maintain a minimum number of edges. Specifically, low-level communication protocols are not sufficient by themselves to achieve high-service availability, especially in the case of ad hoc or dynamic networks with a high degree of node addition and deletion. The concept has been evaluated within a P2P agent application in which each agent has a goal to maintain a preferred number of connections to a number of service providing agents. Using this algorithm, the agents update a weight value associated with each connection, based on the perceived utility of the connection to the corresponding agent. This utility function can be a combination of several node or edge parameters, such as degree k of the target node, or frequency of the message response from the node. This weight is updated using a set of Hebbian-style learning rules, such that the network as a whole exhibits adaptive self-organizing behavior. The principal result is the finding that by limiting the connection neighborhood within the overlay topology, the resulting P2P network can be made highly resilient to targeted attacks on high-degree nodes, while maintaining search efficiency.