Visual Question Answering (VQA) is a fundamental task in computer vision and natural language process fields. Although the “pre-training & finetuning” learning paradigm significantly improves the VQA performance, the adversarial robustness of such a learning paradigm has not been explored. In this paper, we delve into a new problem: using a pre-trained multimodal source model to create adversarial image-text pairs and then transferring them to attack the target VQA models. Correspondingly, we propose a novel VQATTACK model, which can iteratively generate both im- age and text perturbations with the designed modules: the large language model (LLM)-enhanced image attack and the cross-modal joint attack module. At each iteration, the LLM-enhanced image attack module first optimizes the latent representation-based loss to generate feature-level image perturbations. Then it incorporates an LLM to further enhance the image perturbations by optimizing the designed masked answer anti-recovery loss. The cross-modal joint attack module will be triggered at a specific iteration, which updates the image and text perturbations sequentially. Notably, the text perturbation updates are based on both the learned gradients in the word embedding space and word synonym-based substitution. Experimental results on two VQA datasets with five validated models demonstrate the effectiveness of the proposed VQATTACK in the transferable attack setting, compared with state-of-the-art baselines. This work reveals a significant blind spot in the “pre-training & fine-tuning” paradigm on VQA tasks. The source code can be found in the link https://github.com/ericyinyzy/VQAttack.

Pre-trained language models (PLMs) have attracted enormous attention over the past few years with their unparalleled performances. Meanwhile, the soaring cost to train PLMs as well as their amazing generalizability have jointly contributed to few-shot fine-tuning and prompting as the most popular training paradigms for natural language processing (NLP) models. Nevertheless, existing studies have shown that these NLP models can be backdoored such that model behavior is manipulated when trigger tokens are presented. In this paper, we propose PromptFix, a novel backdoor mitigation strategy for NLP models via adversarial prompt-tuning in few-shot settings. Unlike existing NLP backdoor removal methods, which rely on accurate trigger inversion and subsequent model fine-tuning, PromptFix keeps the model parameters intact and only utilizes two extra sets of soft tokens which approximate the trigger and counteract it respectively. The use of soft tokens and adversarial optimization eliminates the need to enumerate possible backdoor configurations and enables an adaptive balance between trigger finding and preservation of performance. Experiments with various backdoor attacks validate the effectiveness of the proposed method and the performances when domain shift is present further shows PromptFix's applicability to models pre-trained on unknown data source which is the common case in prompt tuning scenarios.

Researchers have been studying approaches to steer the behavior of Large Language Models (LLMs) and build personalized LLMs tailored for various applications. While fine-tuning seems to be a direct solution, it requires substantial computational resources and may significantly affect the utility of the original LLM. Recent endeavors have introduced more lightweight strategies, focusing on extracting ``steering vectors'' to guide the model's output toward desired behaviors by adjusting activations within specific layers of the LLM's transformer architecture. However, such steering vectors are directly extracted from the activations of human preference data and thus often lead to suboptimal results and occasional failures, especially in alignment-related scenarios. In this work, we propose an innovative approach that could produce more effective steering vectors through bi-directional preference optimization. Our method is designed to allow steering vectors to directly influence the generation probability of contrastive human preference data pairs, thereby offering a more precise representation of the target behavior. By carefully adjusting the direction and magnitude of the steering vector, we enabled personalized control over the desired behavior across a spectrum of intensities. Extensive experimentation across various open-ended generation tasks, particularly focusing on steering AI personas, has validated the efficacy of our approach. Moreover, we comprehensively investigate critical alignment-concerning scenarios, such as managing truthfulness, mitigating hallucination, and addressing jailbreaking attacks alongside their respective defenses. Remarkably, our method can still demonstrate outstanding steering effectiveness across these scenarios. Furthermore, we showcase the transferability of our steering vectors across different models/LoRAs and highlight the synergistic benefits of applying multiple vectors simultaneously. These findings significantly broaden the practicality and versatility of our proposed method.

In the current Augmented Intelligence of Things (AIoT) vehicular road collaborative edge offloading application scenario, considering the limited computing resources of edge servers and the unpredictability of vehicle user offloading requests, this article fully utilizes the computing resources in the Internet of Vehicles (IoV). During the offloading task execution phase, the caching system is applied to the task execution process to solve the problem of resource waste caused by redundant computing. At the same time, in order to alleviate the problem of low cache hit rate of task calculation results, the task is decomposed into several sub tasks, including vehicle self-execution, edge offloading, cloud offloading, etc. Considering the high dependency between tasks and application services in edge offloading scenarios, as well as the limited storage space of edge servers, this article adopts a cluster edge server application layout strategy. Through the cooperation of various nodes, the offloading services are provided to vehicles as a whole, giving the cluster the ability to handle various tasks. On this basis, in order to reduce the additional system costs caused by task and application misalignment within the cluster, a multiagent deep reinforcement learning algorithm (MADRL) for edge-cloud collaboration is further proposed to obtain the corresponding placement relationship between in vehicle applications and edge servers, and optimize in vehicle computing resources. The simulation results show that the algorithm proposed in this article can improve the success rate of edge offloading, effectively reduce task offloading latency, and improve system resource utilization.

Vision-Language (VL) pre-trained models have shown their superiority on many multimodal tasks. However, the adversarial robustness of such models has not been fully explored. Existing approaches mainly focus on exploring the adversarial robustness under the white-box setting, which is unrealistic. In this paper, we aim to investigate a new yet practical task to craft image and text perturbations using pre-trained VL models to attack black-box fine-tuned models on different downstream tasks. Towards this end, we propose VLATTACK to generate adversarial samples by fusing perturbations of images and texts from both single-modal and multi-modal levels. At the single-modal level, we propose a new block-wise similarity attack (BSA) strategy to learn image perturbations for disrupting universal representations. Besides, we adopt an existing text attack strategy to generate text perturbations independent of the image-modal attack. At the multi-modal level, we design a novel iterative cross-search attack (ICSA) method to update adversarial image-text pairs periodically, starting with the outputs from the single-modal level. We conduct extensive experiments to attack three widely-used VL pretrained models for six tasks on eight datasets. Experimental results show that the proposed VLATTACK framework achieves the highest attack success rates on all tasks compared with state-of-the-art baselines, which reveals a significant blind spot in the deployment of pre-trained VL models.

Recent years have witnessed a surge of certified robust training pipelines against text adversarial perturbation constructed by synonym substitutions. Given a base model, existing pipelines provide prediction certificates either in the discrete word space or the continuous latent space. However, they are isolated from each other with a structural gap. We observe that existing training frameworks need unification to provide stronger certified robustness. Additionally, they mainly focus on building the certification process but neglect to improve the robustness of the base model. To mitigate the aforementioned limitations, we propose a unified framework named UniT that enables us to train flexibly in either fashion by working in the word embedding space. It can provide a stronger robustness guarantee obtained directly from the word embedding space without extra modules. In addition, we introduce the decoupled regularization (DR) loss to improve the robustness of the base model, which includes two separate robustness regularization terms for the feature extraction and classifier modules. Experimental results on widely used text classification datasets further demonstrate the effectiveness of the designed unified framework and the proposed DR loss for improving the certified robust accuracy.

We apply both distance-based and kernel-based mutual dependence measures to independent component analysis (ICA), and generalize dCovICA to MDMICA, minimizing empirical dependence measures as an objective function in both deflation and parallel manners. Solving this minimization problem, we introduce Latin hypercube sampling (LHS), and a global optimization method, Bayesian optimization (BO) to improve the initialization of the Newton-type local optimization method. The performance of MDMICA is evaluated in various simulation studies and an image data example. When the ICA model is correct, MDMICA achieves competitive results compared to existing approaches. When the ICA model is misspecified, the estimated independent components are less mutually dependent than the observed components using MDMICA, while the estimated independent components are prone to be even more mutually dependent than the observed components using other approaches.

Auction-based resource allocation and pricing mechanisms have attracted substantial research interests to enhance the utility gain and fairness of cloud platforms. A fundamental problem in cloud resource auction design that has not been fully addressed is how to ensure the timely execution of applications while allocation decisions have to be made online. Adding to the complexity of designing a truthful and efficient mechanism is that application requests are normally heterogeneous in resource demand and execution requirement. In this paper, we present a novel online combinatorial auction mechanism for deadline- aware multi-resource allocation for cloud platforms, which achieves both strategy-proofness and approximate efficiency on social welfare.

This paper presents the packet-scale paradigm for designing end-to-end congestion control protocols for ultra-high speed networks. The paradigm discards the legacy framework of RTT-scale protocols, and instead builds upon two revolutionary foundations-that of continually probing for available bandwidth at short timescales, and that of adapting the data sending rate so as to avoid overloading the network. Through experimental evaluations with a prototype, we report high performance gains along several dimensions in high-speed networks-the steady-state throughput, adaptability to dynamic cross-traffic, RTT-fairness, and co-existence with the conventional TCP traffic mixes. The paradigm also opens up several issues that are less of a concern for traditional protocols-we summarize our approaches for addressing these.

Efficient wireless channel allocation is becoming a more and more important topic in wireless networking. Dynamic channel allocation is believed to be an effective way to cope with the shortage of wireless channel resource. Up to now, a number of auction mechanisms have been designed to solve the problem of dynamic channel redistribution. Such designs deal with either the problem of single channel allocation or the problem of multiple channels allocation with an assumption of the same per-channel valuation. However, considering the recent outcomes of researches on throughputs of adaptive-width channels and the needs of wireless users in practice, we need to provide buyers with a way to submit various combinatorial bids for channels. This motivates our work on designing a more practical auction mechanism to solve the problem of channel redistribution. In this paper, we propose SPECIAL, which is a Strategy-Proof and EffiCIent multi-channel Auction mechanism for wireLess networks. SPECIAL guarantees the strategy proofness of the channel auction, exploits wireless channels' spatial reusability, and achieves high channel allocation efficiency. Numerical results demonstrate that SPECIAL prevents buyers from manipulating the auction, and achieves high performance.

Multirate opportunistic routing was proposed to achieve high throughput by exploiting multi-user diversity and transmission rate diversity in wireless networks. However, the performance of multirate opportunistic routing still cannot be guaranteed when participating nodes are contributed by different parties and thus have selfish behaviors. In this paper, we present the first Cooperation-Optimal protocol for Multirate Opportunistic routing and forwarding, namely COMO, which guarantee the faithfulness of each player, and thus achieve the social efficiency and strongly Pareto efficient Nash equilibrium with the faithfulness as a given property. Here, social efficiency means that the end-to-end throughput should be maximized, while in a strongly Pareto efficient Nash equilibrium, no one can improve her utility without decreasing the utility of at least one other player. We not only rigorously prove the game-theoretic properties of our incentive protocol, but also extensively evaluate its performance on the ORBIT wireless testbed. Experiment results show that our protocol can prevent participating nodes' selfish behaviors and guarantee high performance of the multirate opportunistic routing protocol with a low communication overhead.

Recently, with the fast development of vehicular ad hoc network (VANET), many more applications on VANET have emerged, such as vehicle navigation systems (VNSs). To make VNSs more effective, a good way is to integrate VNSs with traffic information, which is disseminated over VANET. However, when vehicle users are selfish, they may not participate in the process of traffic information transfer if there is no incentive scheme provided. In this paper, we design a novel incentive scheme to stimulate the vehicles to participate in traffic information transfer. Our analysis shows that under our scheme, the system will converge to a unique subgame perfect equilibrium, an agreement between two vehicle users will be always reached in the first round of the game, and the system will achieve good fairness. Numerical results verify that our incentive scheme can stimulate the vehicles to become willing to participate in traffic information transfer, and the fairness is guaranteed.

Efficient wireless channel allocation is becoming a more and more important topic in wireless networking. Dynamic channel allocation is believed to be an effective way to cope with the shortage of wireless channel resource. In this paper, we propose SPECIAL, which is a Strategy-Proof and EffiCIent multi-channel Auction mechanism for wireLess networks. SPECIAL guarantees the strategy-proofness of the channel auction, exploits wireless channels' spatial reusability, and achieves high channel allocation efficiency.