Visual language models (VLMs) have made significant progress in image captioning tasks, yet recent studies have found they are vulnerable to backdoor attacks. Attackers can inject undetectable perturbations into the data during inference, triggering abnormal behavior and generating malicious captions. These attacks are particularly challenging to detect and defend against due to the stealthiness and cross-modal propagation of the trigger signals. In this paper, we identify two key vulnerabilities by analyzing existing attack patterns: (1) the model exhibits abnormal attention concentration on certain regions of the input image, and (2) backdoor attacks often induce semantic drift and sentence incoherence. Based on these insights, we propose Semantic Reward Defense (SRD), a reinforcement learning framework that mitigates backdoor behavior without requiring any prior knowledge of trigger patterns. SRD learns to apply discrete perturbations to sensitive contextual regions of image inputs via a deep Q-network policy, aiming to confuse attention and disrupt the activation of malicious paths. To guide policy optimization, we design a reward signal named semantic fidelity score, which jointly assesses the semantic consistency and linguistic fluency of the generated captions, encouraging the agent to achieve a robust yet faithful output. SRD offers a trigger-agnostic, policy-interpretable defense paradigm that effectively mitigates local (TrojVLM) and global (Shadowcast) backdoor attacks, reducing ASR to 3.6% and 5.6% respectively, with less than 15% average CIDEr drop on the clean inputs.

The Conditional Sequence Modeling (CSM) paradigm, benefiting from the transformer's powerful distribution modeling capabilities, has demonstrated considerable promise in Reinforcement Learning (RL) tasks. However, much of the work has focused on applying CSM to single online or offline settings, with the general architecture rarely explored. Additionally, existing methods primarily focus on deterministic trajectory modeling, overlooking the randomness of state transitions and the diversity of future trajectory distributions. Fortunately, value-based methods offer a viable solution for CSM, further bridging the potential gap between offline and online RL. In this paper, we propose Value-Guided Decision Transformer (VDT), which leverages value functions to perform advantage-weighting and behavior regularization on the Decision Transformer (DT), guiding the policy toward upper-bound optimal decisions during the offline training phase. In the online tuning phase, VDT further integrates value-based policy improvement with behavior cloning under the CSM architecture through limited interaction and data collection, achieving performance improvement within minimal timesteps. The predictive capability of value functions for future returns is also incorporated into the sampling process. Our method achieves competitive performance on various standard RL benchmarks, providing a feasible solution for developing CSM architectures in general scenarios. Code is available at here.

Abstract Semantic segmentation is a critical topic in computer vision, and it has numerous practical applications, including mobile devices, autonomous driving, and many other fields. However, in these application scenarios, it is often essential for the segmentation models to achieve a balance between efficiency and performance. A lightweight attention‐guided redundancy‐reuse network (LARNet) was proposed to address this challenge in this paper. Specifically, the multi‐scale asymmetric redundancy reuse (MAR) module was designed as the main component of the encoder for dense encoding of contextual semantic features. Furthermore, the efficient attention fusion (EAF) module was established for multi‐scale information fusion via the channel and spatial attention mechanisms in the decoder. A series of experiments were conducted to verify the proposed network. The results of tests on multiple datasets suggest that the network has higher accuracy and faster speed than the existing real‐time semantic segmentation methods.

Real-time semantic segmentation is an important field in computer vision. It is widely employed in real-world scenarios such as mobile devices and autonomous driving, requiring networks to achieve a trade-off between efficiency, performance, and model size. This paper proposes a lightweight network with multi-scale information interaction attention (MSIANet) to solve this issue. Specifically, we designed a multi-scale information interaction module (MSI) is the main component of the encoder and is used to densely encode contextual semantic features. Moreover, we designed the multi-channel attention fusion module (MAF) in the decoder part, thereby realizing multi-scale information fusion through channel attention mechanism and spatial attention mechanism. We verify our method through numerous experiments and prove that our network possesses fewer parameters and faster inference speed compared to most of the existing real-time semantic segmentation methods in multiple datasets.