The widespread deployment of large language models (LLMs) in downstream applications has increased the demand for high-quality system prompts, which have become valuable intellectual assets in the commercial prompt marketplace. Recent studies have demonstrated that system prompts are vulnerable to prompt stealing attacks, where adversaries can extract system prompts from LLM applications by crafting adversarial queries, thereby compromising developers' intellectual property and undermining existing business models. However, prior attack methods suffer from critical limitations including gradient dependency and poor scalability, severely restricting their practical applicability. To address these limitations, we present AGFPS, an automated gradient-free framework that leverages evolutionary optimization to systematically steal prompts. Our approach formulates prompt stealing as a discrete optimization problem, where adversarial queries are modeled as individuals in an evolving population. These individuals are optimized through elite retention, selection, adaptive crossover, and mutation operations. To mitigate local optima convergence, we introduce a progressive fitness evaluation strategy based on adaptive sequence fragmentation that exploits LLMs' autoregressive properties. Comprehensive evaluations across multiple benchmark datasets and mainstream LLMs demonstrates that AGFPS achieves a 95.2% exact system prompt stealing success rate, significantly outperforming manual baselines and surpassing gradient-based methods in 80.6% of scenarios. The generated adversarial queries exhibit remarkable transferability across heterogeneous models and diverse datasets, while maintaining robustness against various defense mechanisms. Our work exposes critical vulnerabilities in current LLM deployment practices and underscores the urgent need for enhanced security measures in LLM applications.

Non-transferable learning (NTL) has emerged as a promising method to protect the intellectual property of deep learning models by restricting cross-domain knowledge transfer. However, the robustness of its transferability constraints against potential attacks has not been explored, especially in practical deployment scenarios. In this paper, we propose a novel black-box attack framework - distribution drift learner (DDL), which effectively bypasses NTL protection mechanisms by only accessing input-output queries of protected models. The theoretical foundation of DDL is derived from the concept of data drift, which takes advantage of the variability of the statistical distribution between the source and target domains. The core innovation of DDL is the integration of distributed perception regularization into a lightweight autoencoder architecture, enabling efficient manipulation of data distribution by optimizing dual objectives (distributed perception loss and reconstruction loss). Training for DDL involves two key steps: First, DDL reconstructs a moderate amount of target domain samples and feeds the reconstructed images into the NTL model to obtain prediction labels. The DDL parameters are then updated by optimizing distributed perception loss and reconstruction loss. Through extensive experiments against standard NTL benchmarks (Digits, CIFAR10, and STL10), we demonstrate that DDL has successfully overcome the barriers of the transferable NTL model and improved the accuracy of the target domain by 81% from 10%. Our work reveals critical vulnerabilities in the NTL framework, particularly with respect to ownership verification and applicability authorization mechanisms, providing valuable insights for developing more robust model protection strategies in real-world applications.

Multimodal Large Language Models (MLLMs) bridge the gap between visual and textual data, enabling a range of advanced applications. However, complex internal interactions among visual elements and their alignment with text can introduce vulnerabilities, which may be exploited to bypass safety mechanisms. To address this, we analyze the relationship between image content and task and find that the complexity of subimages, rather than their content, is key. Building on this insight, we propose the Distraction Hypothesis, followed by a novel framework called Contrasting Subimage Distraction Jailbreaking (CS-DJ), to achieve jailbreaking by disrupting MLLMs alignment through multi-level distraction strategies. CS-DJ consists of two components: structured distraction, achieved through query decomposition that induces a distributional shift by fragmenting harmful prompts into sub-queries, and visual-enhanced distraction, realized by constructing contrasting subimages to disrupt the interactions among visual elements within the model. This dual strategy disperses the model’s attention, reducing its ability to detect and mitigate harmful content. Extensive experiments across five representative scenarios and four popular closed-source MLLMs, including GPT-4o-mini, GPT-4o, GPT-4V, and Gemini-1.5-Flash, demonstrate that CS-DJ achieves average success rates of 52.40% for the attack success rate and 74.10% for the ensemble attack success rate. These results reveal the potential of distraction-based approaches to exploit and bypass MLLMs’ defenses, offering new insights for attack strategies. Our code is available at https://github.com/TeamPigeonLab/CS-DJ.Warning: This paper contains unfiltered content generated by MLLMs that may be offensive to readers

Deep neural network (DNN) models embody the intellectual property of a model owner, as the process of training the DNN model is a complex and resource-intensive task that requires significant investments in data preparation and computing resources. Numerous efforts have been made to protect the intellectual property of DNN models. However, existing methods often come with a critical limitation: they lack robustness, proving effective only in specific intellectual property threat scenarios or they either sacrifice the utility/accuracy of the model owner’s classifier because it interferes with the classifier’s training. To address these issues, we propose GMFIP, a novel generator-based model fingerprinting technology tailored for DNN intellectual property protection. GMFIP stands out for its robustness, extending its utility to various intellectual property threat scenarios rather than specific ones. Furthermore, GMFIP ensures that the utility/accuracy of the model is not affected by protection measures. Specifically, GMFIP begins with the training of the generator, which lays the groundwork for the model fingerprint. The generator generates fingerprints of the unique properties of the source model for verifying model ownership. To further improve the quality of these fingerprints, an extra selection phase dedicated to refining the fingerprints is integrated. Moreover, GMFIP is complemented by a binary classifier, which adapts the threshold setting to get optimal results. Our empirical evaluation includes an ablation study over four state-of-the-art technologies and three image benchmark datasets. Our results demonstrate that GMFIP outperforms other state-of-the-art technologies in effectively distinguishing pirated models from benign models.

Multi-task learning (MTL) supports simultaneous training over multiple related tasks and learns the shared representation. While improving the generalization ability of training on a single task, MTL has higher privacy risk than traditional single-task learning because more sensitive information is extracted and learned in a correlated manner. Unfortunately, very few works have attempted to address the privacy risks posed by MTL. In this article, we first investigate such risk by designing model extraction attack (MEA) and membership inference attack (MIA) in MTL. Then we evaluate the privacy risks on six MTL model architectures and two popular MTL datasets, whose results show that both the number of tasks and the complexity of training data play an important role in the attack performance. Our investigation shows that MTL is more vulnerable than traditional single-task learning under both attacks.

Model extraction attacks (MEAs) allow adversaries to replicate a surrogate model analogous to the target model's decision pattern. While several attacks and defenses have been studied in-depth, the underlying reasons behind our susceptibility to them often remain unclear. Analyzing these implication influence factors helps to promote secure deep learning (DL) systems, it requires studying extraction attacks in various scenarios to determine the success of different attacks and the hallmarks of DLs. However, understanding, implementing, and evaluating even a single attack requires extremely high technical effort, making it impractical to study the vast number of unique extraction attack scenarios. To this end, we present a first-of-its-kind holistic evaluation of implication factors for MEAs which relies on the attack process abstracted from state-of-the-art MEAs. Specifically, we concentrate on four perspectives. we consider the impact of the task accuracy, model architecture, and robustness of the target model on MEAs, as well as the impact of the model architecture of the surrogate model on MEAs. Our empirical evaluation includes an ablation study over sixteen model architectures and four image datasets. Surprisingly, our study shows that improving the robustness of the target model via adversarial training is more vulnerable to model extraction attacks.

With the proliferation of unlabeled data, increasing efforts have been devoted to unsupervised learning. As one of the most representative branches of unsupervised learning, contrastive learning has made great progress with its high efficiency. Unfortunately, privacy threats to contrastive learning have become sophisticated, making it imperative to develop effective technologies that can deal with such threats. To alleviate the privacy issue in contrastive learning, we propose some novel techniques based on differential privacy, which aim at reducing the high sensitivity of gradient in the private training caused by interactive contrastive learning. Specifically, we add differentially private protection to the connection point related to different per-example gradients, which decreases the sensitivity of the gradients significantly. Our experiments on SimCLR and the Barlow Twins show that our approach is superior since it is more accurate while maintaining the same level of privacy protection.

One key factor able to boost the applications of artificial intelligence (AI) in security-sensitive domains is to leverage them responsibly, which is engaged in providing explanations for AI. To date, a plethora of explainable artificial intelligence (XAI) has been proposed to help users interpret model decisions. However, given its data-driven nature, the explanation itself is potentially susceptible to a high risk of exposing privacy. In this paper, we first show that the existing XAI is vulnerable to model extraction attacks and then present an XAI-aware dual-task model extraction attack (DTMEA). DTMEA can attack a target model with explanation services, that is, it can extract both the classification and explanation tasks of the target model. More specifically, the substitution model extracted by DTMEA is a multitask learning architecture, consisting of a sharing layer and two task-specific layers for classification and explanation. To reveal which explanation technologies are more vulnerable to expose privacy information, we conduct an empirical evaluation of four major explanation types in the benchmark data set. Experimental results show that the attack accuracy of DTMEA outperforms the predicted-only method with up to 1.25%, 1.53%, 9.25%, and 7.45% in MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100, respectively. By exposing the potential threats on explanation technologies, our research offers the insights to develop effective tools that are able to trade off security-sensitive relationships.

Malware detection has attracted widespread attention due to the growing malware sophistication. Machine learning based methods have been proposed to find traces of malware by analyzing network traffic. However, network traffic exhibits a series of growing and changing states, which makes it challenging to design a detection model that can detect malicious traffic over a long period without the need for costly retraining. In this paper, we present, IEdroid, an Android malicious network behavior detection method that leverages incremental ensembles for model update. Specifically, we train multiple classifiers to form an interim ensemble in distributed cluster environment, and update the interim ensemble by removing and adding classifiers. The generated model is composed of multiple interim ensembles that can adapt to the network traffic. We evaluated the performance of IEdroid using a dataset consisting of 98,565 benign and 41,267 malicious flows. Results show that IEdroid can effectively detect malicious traffic compared with state-of-the-art detection models. The experiment trained IEdroid on datasets incrementally for 10 times without a significant loss on accuracy, precision, recall, and F-Measure, compared with re-training from scratch with full data.

As machine learning (ML) technologies evolve, various online intelligent services use ML models to provide predictions. Unfortunately, attackers can obtain the private information of the model by interacting with the online service, namely model inversion attack (MIA). However, MIA requires large data sets to be transferred to an online service to obtain the predictive value of the inference model. Besides, the huge transmission may cause the administrator's active defense. To overcome this drawback, we propose a novel MIA scheme, which leverages latent information extracted by an auxiliary neural network as high-dimensional features to simplify what inversion model should learn. The core idea of our scheme is to reuse some parameters of the local pretraining model. Extensive experiments have verified the effectiveness of our method in convolutional neural networks on LFW, pubFig, MNIST data sets. Experimental results show that even with a few queries, our inversion method still work accurately and is superior to other technologies. It is worth mentioning that our method makes it more difficult for administrators to defend against the attack and elicit more investigations for privacy-preserving.

As smartphones carry more and more private information, it has become the main target of malware attacks. Threats on mobile devices have become increasingly sophisticated, making it imperative to develop effective tools that are able to detect and counter such threats. Unfortunately, existing malware detection tools based on machine learning techniques struggle to keep up due to the difficulty in performing online incremental update on the detection models. In this paper, a Two-tier Architecture Malware Detection (TAMD) method is proposed, which can learn from the statistical features of network traffic to detect malware. The first layer of TAMD identifies uncertain samples in the training set through a preliminary classification, whereas the second layer builds an improved classifier by filtering out such samples. We enhance TAMD with an incremental leaning based technique (TAMD-IL), which allows to incrementally update the detection models without retraining it from scratch by removing and adding sub-models in TAMD. We experimentally demonstrate that TAMD outperforms the existing methods with up to 98.72% on precision and 96.57% on recall. We also evaluate TAMD-IL on four concept drift datasets and compare it with classical machine learning algorithms, two state-of-the-art malware detection technologies, and three incremental learning technologies. Experimental results show that TAMD-IL is efficient in terms of both update time and memory usage.