Clustering is a long-standing research problem and a fundamental tool in AI and data analysis. The traditional k-center problem, known as a fundamental theoretical challenge in clustering, has a best possible approximation ratio of 2, and any improvement to a ratio of 2 - ε would imply P = NP. In this work, we study the constrained k-center clustering problem, where instance-level cannot-link (CL) and must-link (ML) constraints are incorporated as background knowledge. Although general CL constraints significantly increase the hardness of approximation, previous work has shown that disjoint CL sets permit constant-factor approximations. However, whether local search can achieve such a guarantee in this setting remains an open question. To this end, we propose a novel local search framework based on a transformation to a dominating matching set problem, achieving the best possible approximation ratio of 2. The experimental results on both real-world and synthetic datasets demonstrate that our algorithm outperforms baselines in solution quality.

Multimodal Large Language Models (MLLMs) extend foundation models to real-world applications by integrating inputs such as text and vision. However, their broad knowledge capacity raises growing concerns about privacy leakage, toxicity mitigation, and intellectual property violations. Machine Unlearning (MU) offers a practical solution by selectively forgetting targeted knowledge while preserving overall model utility. When applied to MLLMs, existing neuron-editing-based MU approaches face two fundamental challenges: (i) forgetting becomes inconsistent across modalities because existing point-wise attribution methods fail to capture the structured, layer-by-layer information flow that connects different modalities; and (ii) general knowledge performance declines when sensitive neurons that also support important reasoning paths are pruned, as this disrupts the model’s ability to generalize. To alleviate these limitations, we propose a multimodal influential neuron path editor (MIP-Editor) for MU. Our approach introduces modality-specific attribution scores to identify influential neuron paths responsible for encoding forget-set knowledge and applies influential-path-aware neuron-editing via representation misdirection. This strategy also enables effective and coordinated forgetting across modalities while preserving the model's general capabilities. Experimental results demonstrate that MIP-Editor achieves a superior unlearning performance on multimodal tasks, with a maximum forgetting rate of 87.75% and up to 54.26% improvement in general knowledge retention. On textual tasks, MIP-Editor achieves up to 80.65% forgetting and preserves 77.90% of general performance.

Large Language Models (LLMs) increasingly underpin intelligent web applications, from chatbots to search and recommendation, where efficient specialization is essential. Low-Rank Adaptation (LoRA) enables such adaptation with minimal overhead, while federated LoRA allows web service providers to fine-tune shared models without data sharing. However, in privacy-sensitive deployments, clients inject varying levels of differential privacy (DP) noise, creating privacy heterogeneity that misaligns individual incentives and global performance. In this paper, we propose WinFLoRA, a privacy-heterogeneous federated LoRA that utilizes aggregation weights as incentives with noise awareness. Specifically, the noises from clients are estimated based on the uploaded LoRA adapters. A larger weight indicates greater influence on the global model and better downstream task performance, rewarding lower-noise contributions. By up-weighting low-noise updates, WinFLoRA improves global accuracy while accommodating clients' heterogeneous privacy requirements. Consequently, WinFLoRA aligns heterogeneous client utility in terms of privacy and downstream performance with global model objectives without third-party involvement. Extensive evaluations demonstrate that across multiple LLMs and datasets, WinFLoRA achieves up to 52.58% higher global accuracy and up to 2.56× client utility than state-of-the-art benchmarks. Source code is publicly available at https://github.com/koums24/WinFLoRA.git.

A reliable autonomous driving system requires a high-precision perception module. Collaborative perception is emerging as a web-scale information-sharing paradigm for autonomous driving, enabling multiple vehicles to collectively achieve a broader perception field than any single vehicle. However, existing approaches necessitate frequent one-to-many communication, which increases network load and leads to information redundancy. This paper presents Octopus, an innovative vehicle-to-road collaboration framework that leverages the computational capabilities of roadside units. Instead of frequent one-to-many communication, vehicles interact only with roadside units, which significantly reduces communication overhead and improves real-time processing efficiency. While this design alleviates communication burdens, vehicles may still struggle to achieve comprehensive situational awareness in highly dynamic environments. To further address this limitation, our framework incorporates global fusion results as prior knowledge, enabling closed-loop fusion to refine vehicle-side perception. Extensive experiments on OPV2V and V2V4Real datasets demonstrate that Octopus excels at collaborative perception, outperforming the state-of-the-art approach up to 11.58% on [email protected], 12.74% on [email protected] and 5514× reduction in communication volume.

We unveil discernible temporal (or historical) trajectory imprints resulting from adversarial example (AE) attacks. Standing in contrast to existing studies, which focus on spatial (or static) imprints within the targeted underlying victim models, we present a novel temporal paradigm for understanding these attacks. These imprints are encapsulated within a single loss metric, spanning universally across diverse tasks such as classification and regression, and modalities including image, text, and audio. Recognizing the distinct nature of loss between adversarial and clean examples, we exploit this temporal imprint for AE detection by proposing (Traceable Adversarial Temporal Imprints). TRAIT operates under minimal assumptions without prior knowledge of attacks, thereby framing the detection challenge as a one-class classification problem. However, detecting AEs is still challenged by significant overlaps between the constructed synthetic losses of adversarial and clean examples due to the absence of ground truth for incoming inputs. TRAIT addresses this challenge by converting the synthetic loss into a spectrum signature, using the technique of Fast Fourier Transform to highlight the discrepancies, drawing inspiration from the temporal nature of the imprints, analogous to time-series signals. Across 12 AE attacks including SMACK (USENIX Sec'2023), TRAIT demonstrates consistent outstanding performance across comprehensively evaluated modalities (image, text, audio), tasks (classification and regression), datasets (nine datasets), and model architectures (e.g., ResNeXt50, BERT, RoBERTa, AudioNet). In all scenarios, TRAIT achieves an AE detection accuracy exceeding 97%, often around 99%, while maintaining a false rejection rate of 1%. TRAIT remains effective under the formulated strong adaptive attacks.

The burgeoning 5G network brings edge servers closer to users to host online applications. These edge servers are typically kept running 24/7 to meet users' computational demands. However, the user coverage, privacy assurance, and service delay have consistently undermined users' confidence, compounded by the significant environmental damage caused by excessive energy consumption. Recently, various approaches have been proposed to tackle the energy-saving demand response issue in the multi-access edge computing (MEC) system. Unfortunately, existing attempts often compromise service quality and energy efficiency for privacy enhancement, and incur significant computational overheads and delays unsuitable for real-time services. Therefore, maintaining satisfying user coverage with energy consumption while adhering to users' privacy demands with low computational overhead is critical to achieving sustainable edge services. To address those challenges, we systematically formulate the location-privacy-preserving edge demand response (LEDR) problem and introduce a novel system named KGEES. KGEES incorporates$k$-anonymity geo-obfuscation to enhance user privacy while leveraging a heuristic approach to finalize resource allocation strategies under geo-distortion greedily to jointly improve system utility, energy, and time efficiency. Comprehensive experiments on a real-world dataset demonstrate that KGEES surpasses the representative approaches by an average of$1.187 \times$in system utility and$1.192 \times$in energy efficiency while being$ 203.5 \times$faster.

While differential privacy (DP) contributes to pre serving data privacy during federated learning (FL), DP-FL suffers from either premature convergence or underutilized privacy budgets and subsequently degraded accuracy. Some recent studies heuristically adjusted the variance of the DP noises but offered no guarantee of optimality, little insight, and limited scalability. This paper presents a new control framework for (ε, δ)-DP FL to address the prevalent issues of DP-FL, i.e., premature convergence or underutilized privacy budgets. The key idea is to interpret the DP perturbation of DP-FL as a control process, where the DP noise variance and communication rounds are interdependent and jointly and adaptively determined. An optimal control framework is proposed to adjust the communication rounds and DP noise variance, adapting to the training accuracy of DP-FL. The optimality gap of (ε, δ)-DP FL is derived under the optimal control framework. The importance of joint orchestration of the DP noise and communication rounds is delineated. Experiments on MLP, CNN, and ResNet-9 models show that, given a privacy level, our control framework allows DP-FL to converge much faster with better accuracy than existing techniques, including those with persistent or heuristically reconfigurable DP noise variances.

Differential privacy (DP) is a leading paradigm for privacy preservation in statistical analysis and learning. Traditional DP mechanisms add noise independently of the original data, which yields inconsistent perturbations across groups and raises fairness concerns in downstream decision and learning tasks. Prior work often assesses fairness via bias and variance, while overlooking noise direction and the scale of the underlying query. We propose a novel Fairly Proportional Noise Mechanism (FPNM) that uniquely considers both the direction and magnitude of noise relative to raw query results. We define mathematical formulations for unfairness, factoring in weighting and temporal decay to allow nonlinear amplification of unfairness. The privacy analysis shows a negative correlation between unfairness and privacy strength that higher privacy levels lead to increased noise and thus greater unfairness. We then generalize to group-level assessment using the average unfairness and the Frobenius norm ($F$-Norm). We also prove that adaptive budget reallocation within a data independent feasible domain preserves the overall$(\epsilon ,\delta )$-DP guarantee. Experiments on both decision and learning tasks demonstrate consistent gains. In decision tasks, the proposed FPNM effectively reduces unfairness, achieving average reductions of 19.17% and 17.32% in$F$-Norm and average unfairness, respectively. In learning tasks, integrating FPNM with DP-SGD achieves fairness comparable to fairness-aware baselines and better accuracy. Besides, empirical privacy remains intact under membership inference attacks. These results highlight its effectiveness in improving utility while preserving privacy, offering a robust and comprehensive approach to enhancing fairness in DP.

Skeleton action recognition models have secured more attention than video-based ones in various applications due to privacy preservation and lower storage requirements. Skeleton data are typically transmitted to cloud servers for action recognition, with results returned to clients via Apps/APIs. However, the vulnerability of skeletal models against adversarial perturbations gradually reveals the unreliability of these systems. Existing black-box attacks all operate in a decision-based manner, resulting in numerous queries that hinder efficiency and feasibility in real-world applications. Moreover, all attacks off the shelf focus on only restricted perturbations, while ignoring model weaknesses when encountered with non-semantic perturbations. In this paper, we propose two query-effIcient Skeletal Adversarial AttaCks, ISAAC-K and ISAAC-N. As a black-box attack, ISAAC-K utilizes Grad-CAM in a surrogate model to extract key joints where minor sparse perturbations are then added to fool the classifier. To guarantee natural adversarial motions, we introduce constraints of both bone length and temporal consistency. ISAAC-K finds stronger adversarial examples on ℓ∞norm, which can encompass those on other norms. Exhaustive experiments substantiate that ISAAC-K can uplift the attack efficiency of the perturbations under 10 skeletal models. Additionally, as a byproduct, ISAAC-N fools the classifier by replacing skeletons unrelated to the action. We surprisingly find that skeletal models are vulnerable to large perturbations where the part-wise non-semantic joints are just replaced, leading to a query-free no-box attack without any prior knowledge. Based on that, four adaptive defenses are eventually proposed to improve the robustness of skeleton recognition models.

Caching web data on edge servers has become a common practice in latency-sensitive services to minimize data retrieval delays for web users. However, the geographic distribution of edge servers and frequent data transmissions make these systems vulnerable to security threats, particularly cache pollution attacks (CPAs). In such attacks, malicious users send excessive requests for unpopular data at abnormal frequencies, causing irrelevant content to be cached and degrading the system’s performance. Traditional CPAs, though impactful in conventional caching systems, are less effective in edge environments where user requests are more diverse and edge servers collaborate in caching strategies. In this paper, we identify a novel attack named data flipping attack (DFA) that targets the data transmission process among edge servers. This attack manipulates request distribution by swapping the frequencies of popular and unpopular data requests, all while maintaining other characteristics like request timing and user identity. This tactic disrupts caching strategies without raising suspicion. Experimental results indicate DFA is independent of user request patterns and demonstrates substantial effectiveness and robustness, successfully forcing edge web users to retrieve data from the cloud across various scales and configurations of edge networks. Furthermore, it evades detection by state-of-the-art methods that rely on specific distribution patterns, such as the Zipf distribution. To counter this attack, we propose an effective defense method that alters the request distribution by frequency distillation, mitigating its impact.

The critical energy infrastructure is undergoing two significant transformations: the rapid increase in renewable distributed energy resources (DER) and the digitalization of the energy sector, collectively shaping what is known as the Internet of Energy (IoE). Artificial intelligence (AI) has become a widely adopted tool for effectively allocating energy and managing sector-related resources, where ensuring fairness is essential. While inherent unfairness in AI systems is well acknowledged, little attention has been given to evaluating this unfairness and its real-world implications within the context of the IoE. In this study, we take a first step to elucidate the unfairness in AI-powered IoE systems induced by malicious users. We introduce Unfairness Score (UScore), a novel metric designed to evaluate the unfairness of machine learning models in real-world IoE scenarios. We then extensively evaluate unfairness attacks using three IoE tabular datasets, demonstrating that AI model fairness can be compromised through data poisoning, whether in centralized learning (CL) or federated learning (FL) settings. Notably, such compromises can occur when malicious users tamper with only a small subset of the data they control. Finally, we propose a novel approach that unifies fairness and differential privacy (DP) by leveraging DP as a provable defense mechanism. This approach provides a universally applicable solution to unfairness attacks, regardless of whether the learning tasks are classification or regression, and is effective in both FL and CL settings. Our contributions represent a significant step in addressing unfairness and privacy concerns in AI-powered IoE systems.

As the 5G network rapidly expands, more edge servers are being deployed to provide more efficient and low-latency mobile services. However, limited edge resources constrain users' demand response, while continuous server operation leads to significant energy consumption, undermining the sustainability of the multi-access edge computing (MEC) system. Existing resource allocation methods rely on accurate user locations, which can lead to privacy exposure, while protection techniques often result in significant service degradation due to spatial distortion. Moreover, user mobility in MEC systems poses new challenges for edge resource management, which requires dynamic server collaboration and user data migration, incurring additional costs and delays. To address these challenges, we propose MoSEEC, which employs user-adaptive differential geo-obfuscation to secure trajectory privacy while dynamically enhancing service performance with energy awareness. Our results demonstrate its superior performance in migration delays and system utility by$1.54 \times$faster and$1.15 \times$higher compared to existing techniques with privacy guarantees, respectively. In addition, our system outperforms state-of-the-art approaches by$5 \times$faster on average in terms of computation overhead.

With large AI systems and models (LAMs) playing an ever-growing role across diverse applications, their impact on the privacy and cybersecurity of critical infrastructure has become a pressing concern. The LAMPS workshop is dedicated to tackling these emerging challenges, promoting dialogue on cutting-edge developments and ethical issues in safeguarding LAMs within critical infrastructure contexts. Bringing together leading experts from around the world, this workshop will delve into the complex privacy and cybersecurity risks posed by LAMs in critical sectors. Attendees will explore innovative solutions, exchange best practices, and contribute to shaping the future research agenda, emphasizing the crucial balance between advancing AI technologies and securing critical digital and physical infrastructures.

Multi-access edge computing (MEC) brings computational resources closer to end-users by widely distributing the physical edge services, reducing end-to-end service latency at the network edge. Continuous operation of edge servers results in high energy usage and significant carbon footprints. Efficient resource management is essential for MEC sustainability. Recent solutions focus on demand response to reduce energy consumption. However, the reduction in available resources due to server shutdowns forces a degradation in the quality of service (QoS) provided to users, significantly impacting their quality of experience (QoE). Moreover, the non-linear relationship between QoS and QoE further complicates the issue. Therefore, maintaining user's QoE with energy consumption is critical to achieving sustainable edge services. To tackle these challenges, we formulate the QoE-aware energy saving (QoEES) problem and propose QESGame, a game-theoretical algorithm to solve this problem effectively and efficiently with a guaranteed convergence to Nash equilibrium. Through extensive evaluations, we demonstrate that QESGame surpasses the representative approaches by up to 20.21%, 41.62%, and 23.54% in terms of the overall system benefit, energy saving, and QoE.

Recent advances in Audio-Language Models (ALMs) have significantly improved multimodal understanding capabilities. However, the introduction of the audio modality also brings new and unique vulnerability vectors. Previous studies have proposed jailbreak attacks that specifically target ALMs, revealing that defenses directly transferred from traditional audio adversarial attacks or text-based Large Language Model (LLM) jailbreaks are largely ineffective against these ALM-specific threats. To address this issue, we propose ALMGuard, the first defense framework tailored to ALMs. Based on the assumption that safety-aligned shortcuts naturally exist in ALMs, we design a method to identify universal Shortcut Activation Perturbations (SAPs) that serve as triggers that activate the safety shortcuts to safeguard ALMs at inference time. To better sift out effective triggers while preserving the model’s utility on benign tasks, we further propose Mel-Gradient Sparse Mask (M-GSM), which restricts perturbations to Mel-frequency bins that are sensitive to jailbreaks but insensitive to speech understanding. Both theoretical analyses and empirical results demonstrate the robustness of our method against both seen and unseen attacks. Overall, \MethodName reduces the average success rate of advanced ALM-specific jailbreak attacks to 4.6\% across four models, while maintaining comparable utility on benign benchmarks, establishing it as the new state of the art. Our code and data are available at https://github.com/WeifeiJin/ALMGuard.

Large Vision-Language Models (LVLMs) have demonstrated impressive capabilities in tasks requiring multimodal understanding. However, recent studies indicate that LVLMs are more vulnerable than LLMs to unsafe inputs and prone to generating harmful content. Existing defense strategies primarily include fine-tuning, input sanitization, and output intervention. Although these approaches provide a certain level of protection, they tend to be resource-intensive and struggle to effectively counter sophisticated attack techniques. To tackle such issues, we propose One-head Defense (Oh Defense), a novel yet simple approach utilizing LVLMs' internal safety capabilities. Through systematic analysis of the attention mechanisms, we discover that LVLMs' safety capabilities are concentrated within specific attention heads that respond differently to safe or unsafe inputs. Further exploration reveals that a single critical attention head can effectively serve as a safety guard, providing a strong discriminative signal that amplifies the model's inherent safety capabilities. Hence, the Oh Defense requires no additional training or external modules, making it computationally efficient while effectively reactivating suppressed safety mechanisms. Extensive experiments across diverse LVLM architectures and unsafe datasets validate our approach, i.e., the Oh Defense achieves near-perfect defense success rates (> 98\%) for unsafe inputs while maintaining low false positive rates (< 5\%) for safe content. The source code is available at https://github.com/AIASLab/Oh-Defense.

Recent advancements in speech synthesis technology have enriched our daily lives, with high-quality and human-like audio widely adopted across real-world applications. However, malicious exploitation like voice-cloning fraud poses severe security risks. Existing defense techniques struggle to address the production large language model (LLM)-based speech synthesis. While previous studies have considered the protection for fine-tuning synthesizers, they assume manually annotated transcripts. Given the labor intensity of manual annotation, end-to-end (E2E) systems leveraging automatic speech recognition (ASR) to generate transcripts are becoming increasingly prevalent, e.g., voice cloning via commercial APIs. Therefore, this E2E speech synthesis also requires new security mechanisms. To tackle these challenges, we propose E2E-VGuard, a proactive defense framework for two emerging threats: (1) production LLM-based speech synthesis, and (2) the novel attack arising from ASR-driven E2E scenarios. Specifically, we employ the encoder ensemble with a feature extractor to protect timbre, while ASR-targeted adversarial examples disrupt pronunciation. Moreover, we incorporate the psychoacoustic model to ensure perturbative imperceptibility. For a comprehensive evaluation, we test 16 open-source synthesizers and 3 commercial APIs across Chinese and English datasets, confirming E2E-VGuard's effectiveness in timbre and pronunciation protection. Real-world deployment validation is also conducted. Our code and demo page are available at https://wxzyd123.github.io/e2e-vguard/.

Differential privacy (DP) is the de facto privacy standard against privacy leakage attacks, including many recently discovered ones against large language models (LLMs). However, we discovered that LLMs could reconstruct the altered/removed privacy from given DP-sanitized prompts. We propose two attacks (black-box and white-box) based on the accessibility to LLMs and show that LLMs could connect the pair of DPsanitized text and the corresponding private training data of LLMs by giving sample text pairs as instructions (in the blackbox attacks) or fine-tuning data (in the white-box attacks). To illustrate our findings, we conduct comprehensive experiments on modern LLMs (e.g., LLaMA-2, LLaMA-3, ChatGPT-3.5, ChatGPT-4, ChatGPT-4o, Claude-3, Claude-3.5, OPT, GPT-Neo, GPT-J, Gemma-2, and Pythia) using commonly used datasets (such as WikiMIA, Pile-CC, and Pile-Wiki) against both wordlevel and sentence-level DP. The experimental results show promising recovery rates, e.g., the black-box attacks against the word-level DP over WikiMIA dataset gave 72.18% on LLaMA2 (70B), 82.39% on LLaMA-3 (70B), 75.35% on Gemma-2, 91.2% on ChatGPT-4o, and 94.01% on Claude-3.5 (Sonnet). More urgently, this study indicates that these well-known LLMs have emerged as a new security risk for existing DP text sanitization approaches in the current environment.

The right to be forgotten mandates that machine learning models enable the erasure of a data owner's data and information from a trained model. Removing data from the dataset alone is inadequate, as machine learning models can memorize information from the training data, increasing the potential privacy risk to users. To address this, multiple machine unlearning techniques have been developed and deployed. Among them, approximate unlearning is a popular solution, but recent studies report that its unlearning effectiveness is not fully guaranteed. Another approach, exact unlearning, tackles this issue by discarding the data and retraining the model from scratch, but at the cost of considerable computational and memory resources. However, not all devices have the capability to perform such retraining. In numerous machine learning applications, such as edge devices, Internet-of-Things (IoT), mobile devices, and satellites, resources are constrained, posing challenges for deploying existing exact unlearning methods. In this study, we propose a Constraint-aware Adaptive Exact Unlearning System at the network Edge (CAUSE), an approach to enabling exact unlearning on resource-constrained devices. Aiming to minimize the retrain overhead by storing sub-models on the resource-constrained device, CAUSE inno-vatively applies a Fibonacci-based replacement strategy and updates the number of shards adaptively in the user-based data partition process. To further improve the effectiveness of memory usage, CAUSE leverages the advantage of model pruning to save memory via compression with minimal accuracy sacrifice. The experimental results demonstrate that CAUSE significantly outperforms other representative systems in realizing exact unlearning on the resource-constrained device by 9.23%-80.86%, 66.21%-83.46%, and 5.26%-194.13% in terms of unlearning speed, energy consumption, and accuracy.

With the rapid adoption of LLM-based code assistants to enhance programming experiences, concerns over extraction attacks targeting private training data have intensified. These attacks specifically aim to extract Personal Information (PI) embedded within the training data of code generation models (CodeLLMs). Existing methods, using either manual or semi-automated techniques, have successfully extracted sensitive data from these CodeLLMs. However, the limited amount of data currently retrieved by extraction attacks risks significantly underestimating the true extent of training data leakage. In this paper, we propose an automatic PI data extraction attack framework against LLM-based code assistants, named Codebreaker. This framework is built on two core components: (i) the introduction of semantic entropy, which evaluates the likelihood of a prompt triggering the model to respond with training data; and (ii) an automatic dynamic mutation mechanism that seamlessly integrates with Codebreaker, reinforcing the iterative process across the framework and promoting greater interconnection between different PI elements within a single response. This boosts reasoning diversity, model memorization, and finally attack performance. Using six series of open-source CodeLLMs (i.e., CodeParrot, StarCoder2, Code Llama, CodeGemma, DeepSeek-Coder, DeepSeek-V3) and two commercial code assistants (i.e., CodeFuse and GPT), we demonstrate the effectiveness of our proposed framework: (i) Codebreaker outperforms all current state-of-the-art extraction attacks by 6.22% ~ 44.9% (averaging 21.79%); (ii) when PI within a single response originates from the same GitHub repository, our framework - considering multiple interconnections in the response - exceeds others by 3.88% ~ 32.37% (averaging 15.31%). Furthermore, we discuss potential defenses, highlighting the urgent need for stronger measures to prevent PI leakage at the base model level.

The unprecedented image generation capability of text-to-image models makes them double-edged swords. While these models allow users to create exquisite images through simple prompts, they also provide adversaries with opportunities to generate Not-Safe-for-Work (NSFW) content, referred to as the jailbreak attack. Despite built-in safety filters serving as a mitigation, their vulnerabilities and associated safety issues remain a significant concern. In this work, we propose MODX, the first modifier-based attack framework for jailbreaking text-to-image models. Modx leverages a heuristic algorithm with two heuristic functions (constraints) to identify modifiers that adjust the artistic genre to subtly introduce unsafe elements that drive the generated images towards NSFW. This approach takes advantage of the fact that filters are unlikely to reject images in certain styles or artistic forms, effectively inducing the models to generate NSFW content. We demonstrate the feasibility of modifier-based jailbreaking with a theoretical analysis, and provide experimental evidence of the effectiveness of MODX. Our results show that MODX outperforms existing methods in successfully achieving jailbreaking across four state-of-the-art text-to-image models. Moreover, we evaluate MODX across additional NSFW categories and on more models or model versions, demonstrating its strong scalability and generalization. Disclaimer: This paper contains NSFW language and imagery that could be offensive, distressing, and/or upsetting. Reader discretion is advised.

Deceptive patterns (DPs) are user interface designs deliberately crafted to manipulate users into unintended decisions, often by exploiting cognitive biases for the benefit of companies or services. While numerous studies have explored ways to identify these deceptive patterns, many existing solutions require significant human intervention and struggle to keep pace with the evolving nature of deceptive designs. To address these challenges, we expanded the deceptive pattern taxonomy from security and privacy perspectives, refining its categories and scope. We created a comprehensive dataset of deceptive patterns by integrating existing small-scale datasets with new samples, resulting in 6,725 images and 10,421 DP instances from mobile apps and websites. We then developed DPGuard, a novel automatic tool leveraging commercial multimodal large language models (MLLMs) for deceptive pattern detection. Experimental results show that DPGuard outperforms state-of-the-art methods. An extensive empirical evaluation on 2,000 popular mobile apps and websites reveals that 25.7% of mobile apps and 49.0% websites feature at least one deceptive pattern instance. Through 4 unexplored case studies that inform security implications, we highlight the critical importance of the unified taxonomy in addressing the growing challenges of Internet deception.

Large-scale models are typically adapted to meet the diverse requirements of model owners and users.However, maintaining multiple specialized versions of the model is inefficient.In response, we propose Aim, a novel model modulation paradigm that enables a single model to exhibit diverse behaviors to meet the specific end requirements.Aim enables two key modulation modes: utility and focus modulations.The former provides model owners with dynamic control over output quality to deliver varying utility levels, and the latter offers users precise control to shift model's focused input features.Aim introduces a logits redistribution strategy that operates in a training data-agnostic and retraining-free manner.We establish a formal foundation to ensure Aim's regulation capability, based on the statistical properties of logits ordering via joint probability distributions.Our evaluation confirms Aim's practicality and versatility for AI model modulation, with tasks spanning image classification, semantic segmentation and text generation, and prevalent architectures including ResNet, SegFormer and Llama.

Protecting sensitive information on data streams is a pivotal challenge for modern systems. Current approaches to providing privacy in data streams can be broadly categorized into two strategies. The first strategy involves transforming the stream into a private sequence of values, enabling the subsequent use of non-private methods of analysis. While effective, this approach incurs high memory costs, often proportional to the size of the database. Alternatively, a compact data structure can be used to provide a private summary of the stream. However, these data structures are limited to predefined queries, restricting their flexibility. To overcome these limitations, we propose a lightweight synthetic data generator, PrivHP, that provides differential privacy guarantees. PrivHP is based on a novel method for the private hierarchical decomposition of the input domain in bounded memory. As the decomposition approximates the cumulative distribution function of the input, it serves as a lightweight structure for synthetic data generation. PrivHP is the first method to provide a principled trade-off between accuracy and space for private hierarchical decompositions. It achieves this by balancing hierarchy depth, noise addition, and selective pruning of low-frequency subdomains while preserving high-frequency ones, all identified in a privacy-preserving manner. To ensure memory efficiency, we employ private sketches to estimate subdomain frequencies without accessing the entire dataset. Central to our approach is the introduction of a pruning parameter k , which enables an almost smooth interpolation between space usage and utility, and a measure of skew tail k , which is a vector of subdomain frequencies containing all but the largest k coordinates. PrivHP processes a dataset X using M = O (k log 2 | X |)) space and, on input domain Ω = [0,1] d , while maintaining ε-differential privacy, produces a synthetic data generator that is at distance O ( M (1-1/d) /ε n + ||tail k ( X )|| 1 /M 1/d n ) from the empirical distribution in the expected Wasserstein metric. Compared to the state-of-the-art, PMM, which achieves accuracy O ((ε n) -1/d ) with memory O (ε n), our method introduces an additional approximation error term of O (||tail k ( X )|| 1 /(M 1/d n)), but operates in significantly reduced space. Additionally, we provide interpretable utility bounds that account for all error sources, including those introduced by the fixed hierarchy depth, privacy noise, hierarchy pruning, and frequency approximations.

AI-enabled critical infrastructures (ACIs) integrate artificial intelligence (AI) technologies into various essential systems and services that are vital to the functioning of society, offering significant implications for efficiency, security and resilience. While adopting decentralized AI approaches (such as federated learning technology) in ACIs is plausible, private and sensitive data are still susceptible to data reconstruction attacks through gradient optimization. In this work, we propose Compressed Differentially Private Aggregation (CDPA), a leakage-resilient, communication-efficient, and carbon-neutral approach for ACI networks. Specifically, CDPA has introduced a novel random bit-flipping mechanism as its primary innovation. This mechanism first converts gradients into a specific binary representation and then selectively flips masked bits with a certain probability. The proposed bit-flipping introduces a larger variance to the noise while providing differentially private protection and commendable efforts in energy savings while applying vector quantization techniques within the context of federated learning. The experimental evaluation indicates that CDPA can reduce communication cost by half while preserving model utility. Moreover, we demonstrate that CDPA can effectively defend against state-of-the-art data reconstruction attacks in both computer vision and natural language processing tasks. We highlight existing benchmarks that generate 2.6x to over 100x more carbon emissions than CDPA. We hope that the CDPA developed in this paper can inform the federated AI-enabled critical infrastructure of a more balanced trade-off between utility and privacy, resilience protection, as well as a better carbon offset with less communication overhead.

The security guarantee of AI-enabled software systems (particularly using deep learning techniques as a functional core) is pivotal against the adversarial attacks exploiting software vulnerabilities. However, little attention has been paid to a systematic investigation of vulnerabilities in such systems. A common situation learned from the open source software community is that deep learning engineers frequently integrate off-the-shelf or open-source learning frameworks into their ecosystems. In this work, we specifically look into deep learning (DL) framework and perform the firstsystematicstudy of vulnerabilities in DL systems through a comprehensive analysis of identified vulnerabilities from Common Vulnerabilities and Exposures (CVE) and open-source DL tools, including TensorFlow, Caffe, OpenCV, Keras, and PyTorch. We propose a two-stream data analysis framework to explore vulnerability patterns from various databases. We investigate the unique DL frameworks and libraries development ecosystems that appear to be decentralized and fragmented. By revisiting the Common Weakness Enumeration (CWE) List, which provides the traditional software vulnerability related practices, we observed that it is more challenging to detect and fix the vulnerabilities throughout the DL systems lifecycle. Moreover, we conducted a large-scale empirical study of 3,049 DL vulnerabilities to better understand the patterns of vulnerability and the challenges in fixing them.

Deep models excel in analyzing image data. However, recent studies on Black-Box Model Inversion (MI) Attacks against image models have revealed the potential to recover concealed (via specific masks) private training images using publicly available images from the same domain as the training data. This study introduces PriDM, a novel diffusion model-based MI attack, illustrating the increased vulnerability of image models. PriDM leverages range-null space decomposition to extract essential range-space information and incorporates it into the diffusion model's sampling process. This enables the recovery of private information from arbitrarily masked images relying solely on images only aligned with the same machine-learning tasks as the target model. To demonstrate PriDM's effectiveness, we conducted experiments with various adversary background knowledge, including different public dataset domains and image masks. Results show PriDM produces recovered images of significantly higher quality, approximately twice as good as existing methods. Moreover, in scenarios involving complex backgrounds, PriDM outperforms the state-of-the-art by approximately 70%. In specific background knowledge scenarios, such as compressed and blurred images, our method achieves an almost 100% success rate. Additionally, PriDM performs well with real-world background knowledge including individuals wearing masks and randomly masked face images, which are not considered by existing works.

The widespread adoption of 5 G networks and mobile devices has led to a surge in the generation of private data, creating massive data streams. Securing and continuously releasing histogram data over sliding windows in these streams has become a critical issue, as it enables understanding recent collective phenomena in data streams while preserving individual privacy. Existing state-of-the-art methods require buffering all data from each sliding window to reconstruct accurate histograms, which is unnecessary and significantly hampers efficiency. This paper proposes an online streaming sampling publication framework with differential privacy, named thePublishingApproach withSliding window estimation-count sketch(PAS), which constructs an approximate histogram without buffering each sliding window and subsequently generates publishable histograms. Specifically, we introduce a novel memory-efficient sketch structure called theSliding WindowEstimation-CountSketch(SES), which facilitates rapid retrieval of counts within sliding window intervals while providing guaranteed data protection. The output of this sketch structure approximates true counts while theoretically incorporating differentially private noise, thus ensuring$(\epsilon , \delta )$-differential privacy. Moreover, to improve the speed of histogram generation and reduce processing time in PAS, we propose an adaptive histogram generation algorithm based on SES. Extensive experiments are conducted to demonstrate the effectiveness of the proposed methods in comparison with other publication methods.

Face authentication systems have brought significant convenience and advanced developments, yet they have become unreliable due to their sensitivity to inconspicuous perturbations, such as adversarial attacks. Existing defenses often exhibit weaknesses when facing various attack algorithms and adaptive attacks or compromise accuracy for enhanced security. To address these challenges, we have developed a novel and highly efficient non-deep-learning-based image filter called the Iterative Window Mean Filter (IWMF) and proposed a new framework for adversarial purification, named IWMF-Diff, which integrates IWMF and denoising diffusion models. These methods can function as pre-processing modules to eliminate adversarial perturbations without necessitating further modifications or retraining of the target system. We demonstrate that our proposed methodologies fulfill four critical requirements: preserved accuracy, improved security, generalizability to various threats in different settings, and better resistance to adaptive attacks. This performance surpasses that of the state-of-the-art adversarial purification method, DiffPure. Our code is released athttps://github.com/azrealwang/iwmfdiff.

As one of the most successful techniques in hunting software bugs, Coverage-guided Greybox Fuzzing (CGF) intends to move fuzzing campaigns towards executions that can trigger bugs. This process can be divided into two steps, including reaching suspicious code regions and exploring their execution states. Many CGFs propose approaches to efficiently reach suspicious code regions and individual execution states, but fail to explore complex execution states. The challenge is how to explore execution states so that fuzzing can detect multiple types of bugs, while maintaining the code coverage. To address this challenge, we proposeBazzAFLto investigate code coverage and multiple types of bugs. The crux ofBazzAFLis to maintain a bunch of seed groups, where each seed saves the best performance on one objective. With the seed group,BazzAFLprioritizes code regions that most likely contain bugs based on multi-objective optimization and adaptively divides energy among the seeds in a group based on Shannon's entropy. Meanwhile, during mutation,BazzAFLtends to mutating the bytes that can change the execution states. With these solutions,BazzAFLgradually moves fuzzing campaigns towards locations and execution states of bugs. Experimental results show thatBazzAFLidentifies at least 62 more bugs on 24 programs compared with other fuzzers.

Model extraction attacks currently pose a non-negligible threat to the security and privacy of deep learning models. By querying the model with a small dataset and using the query results as the ground-truth labels, an adversary can steal a piracy model with performance comparable to the original model. Two key issues that cause the threat are, on the one hand, accurate and unlimited queries can be obtained by the adversary; on the other hand, the adversary can aggregate the query results to train the model step by step. The existing defenses usually employ model watermarking or fingerprinting to protect the ownership. However, these methods cannot proactively prevent the violation from happening. To mitigate the threat, we propose QUEEN (QUEry unlEarNing) that proactively launches counterattacks on potential model extraction attacks from the very beginning. To limit the potential threat, QUEEN has sensitivity measurement and outputs perturbation that prevents the adversary from training a piracy model with high performance. In sensitivity measurement, QUEEN measures the single query sensitivity by its distance from the center of its cluster in the feature space. To reduce the learning accuracy of attacks, for the highly sensitive query batch, QUEEN applies query unlearning, which is implemented by gradient reverse to perturb the softmax output such that the piracy model will generate reverse gradients to worsen its performance unconsciously. Experiments show that QUEEN outperforms the state-of-the-art defenses against various model extraction attacks with a relatively low cost to the model accuracy. The artifact is publicly available athttps://github.com/MaraPapMann/QUEEN.

In contrast to prevalent Federated Learning (FL) privacy inference techniques such as generative adversarial networks attacks, membership inference attacks, property inference attacks, and model inversion attacks, we devise an innovative privacy threat: the Data Distribution Decompose Attack on FL, termedDecaf. This attack enables an honest-but-curious FL server to meticulously profile the proportion of each class owned by the victim FL user, divulging sensitive information like local market item distribution and business competitiveness. The crux ofDecaflies in the profound observation that the magnitude of local model gradient changes closely mirrors the underlying data distribution, including the proportion of each class.Decafaddresses two crucial challenges: accurately identify the missing/null class(es) given by any victim user as a premise and then quantify the precise relationship between gradient changes and each remaining non-null class. Notably,Decafoperates stealthily, rendering it entirely passive and undetectable to victim users regarding the infringement of their data distribution privacy. Experimental validation on five benchmark datasets (MNIST, FASHION-MNIST, CIFAR-10, FER-2013, and SkinCancer) employing diverse model architectures, including customized convolutional networks, standardized VGG16, and ResNet18, demonstratesDecaf’s efficacy. Results indicate its ability to accurately decompose local user data distribution, regardless of whether it is IID or non-IID distributed. Specifically, the dissimilarity measured using$L_{\infty }$distance between the distribution decomposed byDecafand ground truth is consistently below 5% when no null classes exist. Moreover,Decafachieves 100% accuracy in determining any victim user’s null classes, validated through formal proof.

Critical infrastructures are increasingly integrating artificial intelligence (AI) technologies, including large language models (LLMs), into essential systems and services that are vital to societal functioning. Fine-tuning LLMs for specific domain tasks are crucial for their effective deployment in these contexts, but this process must carefully address both privacy and security concerns. Without proper safeguards, such integration can introduce additional risks, such as data leakage during training and diminished model trustworthiness due to the need for model compression to operate within limited bandwidth and computational capacity constraints. In this paper, we proposeHardening LLM Fine-tuning framework(HARDLLM), which addresses these challenges through two key components: (i) we develop a differentially private data selection method that ensures privacy protection by training the model exclusively on sampled and synthesized public data, thereby preventing any direct use of private data and enhancing leakage resilience throughout the training process, and (ii) we introduce a trustworthiness-aware model quantization approach to improve LLMs performance, such as reducing toxicity, enhancing adversarial robustness, and mitigating stereotypes, while maintaining negligible impact on model utility. Experimental results show that, the proposed algorithm ensures differential privacy when privacy budget is set at ϵ = 0.5, with only a 1% drop in accuracy, while other state-of-the-art methods experience an accuracy drop of at least 20% under the same privacy budget. Additionally, our quantization approach improves the trustworthiness of fine-tuned LLMs by an average of 3-4%, with only a negligible utility loss (approximately 1%) at a 50% compression rate.

Model inversion (MI) attacks, for which effective defense strategies are still lacking, pose significant risks to privacy by reconstructing private training data through access to well-trained classifiers. Addressing this concern, this study introduces TrapNet, designed to defend against advanced MI attacks while maintaining good model utility. TrapNet intentionally injects trapdoors into the classification manifold of the protected target model. In this way, TrapNet can effectively mislead MI attack optimization. Specifically, TrapNet leverages a conditional GAN (cGAN) trained on the private dataset to generate diverse and realistic trapdoor samples. In addition, we propose a graph-matching self-obfuscation strategy and an entropy regularization technique to optimize trapdoor injection while preserving model utility. Compared to the existing defense, TrapNet can provide universal protection to all target classes without access to any auxiliary public data. Extensive experiments on CelebA, VGG-Face, and VGG-Face2 datasets demonstrate TrapNet’s superior performance over existing defenses, including the most advanced NetGuard and BiDO, against state-of-the-art model inversion attacks, i.e., PLG-MI, LOMMA, and Plug&Play.

Differential Privacy (DP) is a preeminent technique for data privacy by introducing noise to sensitive information. However, traditional DP mechanisms excessively rely on third parties to ensure traceability, necessitating strong background assumptions that are frequently impractical in real-world scenarios. This reliance makes it difficult to preserve both privacy and traceability. To address these challenges, we propose a novel Traceable and Collision-Resilient Differential Privacy (TCRDP) mechanism. The TCRDP mechanism simultaneously publishes perturbed results and data fingerprints, retaining partial information from the original data in a collision-resilient manner to facilitate future verification. Moreover, the TCRDP mechanism integrates an innovative noise generation process, leveraging hash values and a customized Laplace-like distribution to produce noise. This strategy mitigates the risk of adversaries compromising privacy through enumeration and yields a more concentrated noise distribution with reduced variance. We evaluated the TCRDP mechanism using three datasets: ICUs, Diabetes, and RAHRD, across various query types. The experimental results demonstrated significant improvements in data utility, with the TCRDP mechanism achieving great reductions in Mean Absolute Error (MAE) and Mean Squared Error (MSE) compared to traditional mechanisms. The TCRDP mechanism also maintained lower Accuracy Loss (AL) across different privacy budgets and dataset sizes, highlighting its robustness and scalability. These findings underscore the potential of the TCRDP mechanism to advance privacy-preserving data analysis, offering significant enhancements over existing methods in both accuracy and utility.

The right to be forgotten has incentivized machine unlearning, but a key challenge persists: the lack of reliable methods to verify unlearning conducted by model providers. This gap facilitates dishonest model providers to deceive data contributors. Current approaches often rely on invasive methods like backdoor injection. However, it poses security concerns and is also inapplicable to legacy data—already released data. To tackle this challenge, this work initializes the first non-invasive unlearning verification framework which operates at triple-granularity (class-, volume-, sample-level) to assess the data facticity and volume integrity of machine unlearning. In this paper, we propose a framework, named TruVRF, encompasses three Unlearning-Metrics, each tailored to counter different types of dishonest model providers or servers (Neglecting Server, Lazy Server, Deceiving Server). TruVRF leverages non-invasive model sensitivity to enable multi-granularity verification of unlearning. Specifically, Unlearning-Metric-I checks if the removed class matches the data contributor’s unlearning request, Unlearning-Metric-II measures the amount of unlearned data, and Unlearning-Metric-III validates the correspondence of a specific unlearned sample with the requested deletion. We conducted extensive evaluations of TruVRF efficacy across three datasets, and notably, we also evaluated the effectiveness and computational overhead of TruVRF in real-world applications for the face recognition dataset. Our experimental results demonstrate that TruVRF achieves robust verification performance: Unlearning-Metric-I and -III achieve over 90% verification accuracy on average against dishonest servers, while Unlearning-Metric-II maintains an inference deviation within 4.8% to 8.2%. Additionally, TruVRF demonstrates generalizability across diverse conditions, including varying numbers of unlearned classes and sample volumes. Significantly, TruVRF is applied to two state-of-theart unlearning frameworks: SISA [3] (presented at Oakland’21) and Amnesiac Unlearning [18], representing exact and approximate unlearning methods, respectively, which affirm TruVRF’s practicality. In addition, we conducted extensive evaluations around TruVRF, including ablation experiments, trade-offs in computational overhead, and the robustness of model sensitivity, among others.

Many practical applications impose a new challenge of utilizing instance-level background knowledge (e.g., subsets of similar or dissimilar data points) within their input data to improve clustering results. In this work, we build on the widely adopted k-center clustering, modeling its input instance-level background knowledge as must-link (ML) and cannot-link (CL) constraint sets, and formulate the constrained k-center problem. Given the long-standing challenge of developing efficient algorithms for constrained clustering problems, we first derive an efficient approximation algorithm for constrained k-center at the best possible approximation ratio of 2 with linear programming (LP)-rounding technology. Recognizing the limitations of LP-rounding algorithms including high runtime complexity and challenges in parallelization, we subsequently develop a greedy algorithm that does not rely on the LP and can be efficiently parallelized. This algorithm also achieves the same approximation ratio 2 but with lower runtime complexity. Lastly, we empirically evaluate our approximation algorithm against baselines on various real datasets, validating our theoretical findings and demonstrating significant advantages of our algorithm in terms of clustering cost, quality, and runtime complexity.

Video recognition systems are vulnerable to adversarial examples. Recent studies show that style transfer-based and patch-based unrestricted perturbations can effectively improve attack efficiency. These attacks, however, face two main challenges: 1) Adding large stylized perturbations to all pixels reduces the naturalness of the video and such perturbations can be easily detected. 2) Patch-based video attacks are not extensible to targeted attacks due to the limited search space of reinforcement learning that has been widely used in video attacks recently. In this paper, we focus on the video black-box setting and propose a novel attack framework named LogoStyleFool by adding a stylized logo to the clean video. We separate the attack into three stages: style reference selection, reinforcement-learning-based logo style transfer, and perturbation optimization. We solve the first challenge by scaling down the perturbation range to a regional logo, while the second challenge is addressed by complementing an optimization stage after reinforcement learning. Experimental results substantiate the overall superiority of LogoStyleFool over three state-of-the-art patch-based attacks in terms of attack performance and semantic preservation. Meanwhile, LogoStyleFool still maintains its performance against two existing patch-based defense methods. We believe that our research is beneficial in increasing the attention of the security community to such subregional style transfer attacks.

Center-based clustering has attracted significant research interest from both theory and practice. In many practical applications, input data often contain background knowledge that can be used to improve clustering results. In this work, we build on widely adopted k-center clustering and model its input background knowledge as must-link (ML) and cannot-link (CL) constraint sets. However, most clustering problems including k-center are inherently NP-hard, while the more complex constrained variants are known to suffer severer approximation and computation barriers that significantly limit their applicability. By employing a suite of techniques including reverse dominating sets, linear programming (LP) integral polyhedron, and LP duality, we arrive at the first efficient approximation algorithm for constrained k-center with the best possible ratio of 2. We also construct competitive baseline algorithms and empirically evaluate our approximation algorithm against them on a variety of real datasets. The results validate our theoretical findings and demonstrate the great advantages of our algorithm in terms of clustering cost, clustering quality, and running time.

To better understand the output of deep neural networks (DNN), attribution based methods have been an important approach for model interpretability, which assign a score for each input dimension to indicate its importance towards the model outcome. Notably, the attribution methods use the ax- ioms of sensitivity and implementation invariance to ensure the validity and reliability of attribution results. Yet, the ex- isting attribution methods present challenges for effective in- terpretation and efficient computation. In this work, we in- troduce MFABA, an attribution algorithm that adheres to ax- ioms, as a novel method for interpreting DNN. Addition- ally, we provide the theoretical proof and in-depth analy- sis for MFABA algorithm, and conduct a large scale exper- iment. The results demonstrate its superiority by achieving over 101.5142 times faster speed than the state-of-the-art at- tribution algorithms. The effectiveness of MFABA is thor- oughly evaluated through the statistical analysis in compar- ison to other methods, and the full implementation package is open-source at: https://github.com/LMBTough/MFABA.

With large AI systems and models (LAMs) playing an ever-growing role across diverse applications, their impact on the privacy and cybersecurity of critical infrastructure has become a pressing concern. The LAMPS workshop is dedicated to tackling these emerging challenges, promoting dialogue on cutting-edge developments and ethical issues in safeguarding LAMs within critical infrastructure contexts. Bringing together leading experts from around the world, this workshop will delve into the complex privacy and cybersecurity risks posed by LAMs in critical sectors. Attendees will explore innovative solutions, exchange best practices, and contribute to shaping the future research agenda, emphasizing the crucial balance between advancing AI technologies and securing critical digital and physical infrastructures.

All current backdoor attacks on deep learning (DL) models fall under the category of a vertical class backdoor (VCB).In VCB attacks, any sample from a class activates the implanted backdoor when the secret trigger is present, regardless of whether it is a sub-type source-class-agnostic backdoor or a source-class-specific backdoor. For example, a trigger of sunglasses could mislead a facial recognition model when either an arbitrary (source-class-agnostic) or a specific (source-class-specific) person wears sunglasses. Existing defense strategiesoverwhelmingly focus on countering VCB attacks, especially those that are source-class-agnostic. This narrow focus neglects the potential threat of other simpler yet general backdoor types, leading to false security implications. It is, therefore, crucial to discover and elucidate unknown backdoor types, particularly those that can be easily implemented, as a mandatory step before developing countermeasures.

Out-of-distribution (OOD) detection aims to detect testing samples far away from the in-distribution (ID) training data, which is crucial for the safe deployment of machine learning models in the real world. Distance-based OOD detection methods have emerged with enhanced deep representation learning. They identify unseen OOD samples by measuring their distances from ID class centroids or prototypes. However, existing approaches learn the representation relying on oversimplified data assumptions, e.g. modeling ID data of each class with one centroid class prototype or using loss functions not designed for OOD detection, which overlook the natural diversities within the data. Naively enforcing data samples of each class to be compact around only one prototype leads to inadequate modeling of realistic data and limited performance. To tackle these issues, we propose PrototypicAl Learning with a Mixture of prototypes (PALM) that models each class with multiple prototypes to capture the sample diversities, which learns more faithful and compact samples embeddings for enhanching OOD detection. Our method automatically identifies and dynamically updates prototypes, assigning each sample to a subset of prototypes via reciprocal neighbor soft assignment weights. To learn embeddings with multiple prototypes, PALM optimizes a maximum likelihood estimation (MLE) loss to encourage the sample embeddings to compact around the associated prototypes, as well as a contrastive loss on all prototypes to enhance intra-class compactness and inter-class discrimination at the prototype level. Compared to previous methods with prototypes, the proposed mixture prototype modeling of PALM promotes the representations of each ID class to be more compact and separable from others and the unseen OOD samples, resulting in more reliable OOD detection. Moreover, the automatic estimation of prototypes enables our approach to be extended to the challenging OOD detection task with unlabelled ID data. Extensive experiments demonstrate the superiority of PALM over previous methods, achieving state-of-the-art average AUROC performance of 93.82 on the challenging CIFAR-100 benchmark.

Due to the real-world noise and human-added perturbations, attaining the trustworthiness of deep neural networks (DNNs) is a challenging task. Therefore, it becomes essential to offer explanations for the decisions made by these non-linear and complex parameterized models. Attribution methods are promising for this goal, yet its performance can be further improved. In this paper, for the first time, we present that the decision boundary exploration approaches of attribution are consistent with the process for transferable adversarial attacks. Specifically, the transferable adversarial attacks craft general adversarial samples from the source model, which is consistent with the generation of adversarial samples that can cross multiple decision boundaries in attribution. Utilizing this consistency, we introduce a novel attribution method via model parameter exploration. Furthermore, inspired by the capability of frequency exploration to investigate the model parameters, we provide enhanced explainability for DNNs by manipulating the input features based on frequency information to explore the decision boundaries of different models. Large-scale experiments demonstrate that our \textbf{A}ttribution method for \textbf{E}xplanation with model parameter e\textbf{X}ploration (AttEXplore) outperforms other state-of-the-art interpretability methods. Moreover, by employing other transferable attack techniques, AttEXplore can explore potential variations in attribution outcomes. Our code is available at: https://github.com/LMBTough/ATTEXPLORE.

Randomized Smoothing (RS) is currently a scalable certified defense method providing robustness certification against adversarial examples. Although significant progress has been achieved in providing defenses against $\ell_p$ adversaries, the interaction between the smoothing distribution and the robustness certification still remains vague. In this work, we comprehensively study the effect of two families of distributions, named Exponential Standard Gaussian (ESG) and Exponential General Gaussian (EGG) distributions, on Randomized Smoothing and Double Sampling Randomized Smoothing (DSRS). We derive an analytic formula for ESG’s certified radius, which converges to the origin formula of RS as the dimension $d$ increases. Additionally, we prove that EGG can provide tighter constant factors than DSRS in providing $\Omega(\sqrt{d})$ lower bounds of $\ell_2$ certified radius, and thus further addresses the curse of dimensionality in RS. Our experiments on real-world datasets confirm our theoretical analysis of the ESG distributions, that they provide almost the same certification under different exponents $\eta$ for both RS and DSRS. In addition, EGG brings a significant improvement to the DSRS certification, but the mechanism can be different when the classifier properties are different. Compared to the primitive DSRS, the increase in certified accuracy provided by EGG is prominent, up to 6.4% on ImageNet.

Deep neural networks (DNNs) have achieved state-of-the-art performance across various applications. However, ensuring the reliability and trustworthiness of DNNs requires enhanced interpretability of model inputs and outputs. As an effective means of Explainable Artificial Intelligence (XAI) research, the interpretability of existing attribution algorithms varies depending on the choice of reference point, the quality of adversarial samples, or the applicability of gradient constraints in specific tasks. To thoroughly explore the attribution integration paths, in this paper, inspired by the iterative generation of high-quality samples in the diffusion model, we propose an Iterative Search Attribution (ISA) method. To enhance attribution accuracy, ISA distinguishes the importance of samples during gradient ascent and descent, while clipping the relatively unimportant features in the model. Specifically, we introduce a scale parameter during the iterative process to ensure the features in next iteration are always more significant than those in current iteration. Comprehensive experimental results show that our method has superior interpretability in image recognition tasks compared with state-of-the-art baselines. Our code is available at: https://github.com/LMBTough/ISA

Deep Neural Network (DNN) models are often deployed in resource-sharing clouds as Machine Learning as a Service (MLaaS) to provide inference services. To steal model architectures that are of valuable intellectual properties, a class of attacks has been proposed via different side-channel leakage, posing a serious security challenge to MLaaS.Also targeting MLaaS, we propose a new end-to-end attack, DeepTheft, to accurately recover complex DNN model architectures on general processors via the RAPL (Running Average Power Limit)-based power side channel. While unprivileged access to the RAPL has been disabled in bare-metal OSes, we observe that the RAPL is still legitimately accessible in a platform as a service, e.g., the latest docker environment of version 20.10.18 used in this work. However, an attacker can acquire only a low sampling rate (1 KHz) of the time-series energy traces from the RAPL interface, rendering existing techniques ineffective in stealing large and deep DNN models. To this end, we design a novel and generic learning-based framework consisting of a set of meta-models, based on which DeepTheft is demonstrated to have high accuracy in recovering a large number (thousands) of models architectures from different model families including the deepest ResNet152. Particularly, DeepTheft has achieved a Levenshtein Distance Accuracy of 99.75% in recovering network structures, and a weighted average F1 score of 99.60% in recovering diverse layer-wise hyperparameters. Besides, our proposed learning framework is general to other time-series side-channel signals. To validate its generalization, another existing side channel is exploited, i.e., CPU frequency. Different from RAPL, CPU frequency is accessible to unprivileged users in bare-metal OSes. By using our generic learning framework trained against CPU frequency traces, DeepTheft has shown similarly high attack performance in stealing model architectures.

Machine unlearning has become a promising solution for fulfilling the "right to be forgotten", under which individuals can request the deletion of their data from machine learning models. However, existing studies of machine unlearning mainly focus on the efficacy and efficiency of unlearning methods, while neglecting the investigation of the privacy vulnerability during the unlearning process. With two versions of a model available to an adversary, that is, the original model and the unlearned model, machine unlearning opens up a new attack surface. In this paper, we conduct the first investigation to understand the extent to which machine unlearning can leak the confidential content of the unlearned data. Specifically, under the Machine Learning as a Service setting, we propose unlearning inversion attacks that can reveal the feature and label information of an unlearned sample by only accessing the original and unlearned model. The effectiveness of the proposed unlearning inversion attacks is evaluated through extensive experiments on benchmark datasets across various model architectures and on both exact and approximate representative unlearning approaches. The experimental results indicate that the proposed attack can reveal the sensitive information of the unlearned data. As such, we identify three possible defenses that help to mitigate the proposed attacks, while at the cost of reducing the utility of the unlearned model. The study in this paper uncovers an underexplored gap between machine unlearning and the privacy of unlearned data, highlighting the need for the careful design of mechanisms for implementing unlearning without leaking the information of the unlearned data.

The susceptibility of Deep Neural Networks (DNNs) to adversarial attacks and their limited robustness to real-world variations pose substantial challenges to their widespread adoption. Adversarial training has shown promise in fortifying models against such perturbations, however current methods are often specific to a single type of attack and can significantly diminish the model’s overall performance. In response, we present LAtent Concept Masking for robUStness (LACMUS), a novel perceptually-driven methodology that enhances DNN robustness without requiring prior knowledge about the adversarial contexts. We argue that DNNs’ sensitivity to adversarial perturbations and distribution drifts stems from overfitting to non-common concepts within the dataset, leading to an over-reliance on specific learned instances and increased vulnerability. LACMUS addresses this by mapping high-dimensional data into a latent conceptual space to identify and navigate patterns of "non-common concepts" within the latent concept space. It then applies a concept masking strategy to selectively obscure data features, prompting the model to base its decisions on a wider array of information and thus enhancing its decision-making robustness. LACMUS distinguishes itself as a versatile, attack-agnostic framework that employs concept-wise augmentation to enhance robustness against a spectrum of adversarial, semantic, and distributional challenges. Our contributions include the development of a tool for robustness enhancement, a mechanism for mapping data to latent concept space, a strategy for identifying patterns of concept-wise misclassification, and a novel data augmentation module that leverages latent concepts. LACMUS is proven to enhance model resilience and generalization, even when training data is scarce, with experiments on MNIST, CIFAR-10, ImageNet, and CelebA supporting its effectiveness. We also provide augmented datasets to the research community, bolstering the robustness of models trained on them.

The growing complexity of deep neural network models in modern application domains necessitates a complex training process that involves extensive data, sophisticated design, and substantial computation. The trained model inherently encapsulates the intellectual property owned by the model developer (or the model owner). Consequently, safeguarding the model from unauthorized use by entities who obtain access to the model (or the model controllers), i.e., preserving the fundamental rights and proprietary interests of the model owner, has become a critical necessity.In this work, we propose CORELOCKER, employing the strategic extraction of a small subset of significant weights from the neural network. This subset serves as the access key to unlock the model’s complete capability. The extraction of the key can be customized to varying levels of utility that the model owner intends to release. Authorized users with the access key have full access to the model, while unauthorized users can have access to only part of its capability. We establish a formal foundation to underpin CORELOCKER, which provides crucial lower and upper bounds for the utility disparity between pre- and post-protected networks. We evaluate CORELOCKER using representative datasets such as Fashion-MNIST, CIFAR-10, and CIFAR-100, as well as real-world models including Vg-gNet, ResNet, and DenseNet. Our experimental results confirm its efficacy. We also demonstrate CORELOCKER’s resilience against advanced model restoration attacks based on fine-tuning and pruning.

The deployment of Graph Neural Networks (GNNs) within Machine Learning as a Service (MLaaS) has opened up new attack surfaces and an escalation in security concerns regarding model-centric attacks. These attacks can directly manipulate the GNN model parameters during serving, causing incorrect predictions and posing substantial threats to essential GNN applications. Traditional integrity verification methods falter in this context due to the limitations imposed by MLaaS and the distinct characteristics of GNN models.In this research, we introduce a groundbreaking approach to protect GNN models in MLaaS from model-centric attacks. Our approach includes a comprehensive verification schema for GNN’s integrity, taking into account both transductive and inductive GNNs, and accommodating varying pre-deployment knowledge of the models. We propose a query-based verification technique, fortified with innovative node fingerprint generation algorithms. To deal with advanced attackers who know our mechanisms in advance, we introduce randomized fingerprint nodes within our design. The experimental evaluation demonstrates that our method can detect five representative adversarial model-centric attacks, displaying 2 to 4 times greater efficiency compared to baselines.

The objective of differential privacy (DP) is to protect privacy by producing an output distribution that is indistinguishable between any two neighboring databases. However, traditional differentially private mechanisms tend to produce unbounded outputs in order to achieve maximum disturbance range, which is not always in line with real-world applications. Existing solutions attempt to address this issue by employing post-processing or truncation techniques to restrict the output results, but at the cost of introducing bias issues. In this paper, we propose a novel differentially private mechanism which uses a composite probability density function to generate bounded and unbiased outputs for any numerical input data. The composition consists of an activation function and a base function, providing users with the flexibility to define the functions according to the DP constraints. We also develop an optimization algorithm that enables the iterative search for the optimal hyper-parameter setting without the need for repeated experiments, which prevents additional privacy overhead. Furthermore, we evaluate the utility of the proposed mechanism by assessing the variance of the composite probability density function and introducing two alternative metrics that are simpler to compute than variance estimation. Our extensive evaluation on three benchmark datasets demonstrates consistent and significant improvement over the traditional Laplace and Gaussian mechanisms. The proposed bounded and unbiased composite differentially private mechanism will underpin the broader DP arsenal and foster future privacy-preserving studies.

The global deployment of the 5G network has led to a substantial increase in the deployment of edge servers to host web applications, catering to the growing demand for low service latency by edge web users. Yet, running edge servers 24/7 leads to enormous energy consumption and excessive carbon emissions. Energy-efficient edge resource provision is desired to achieve sustainable development goals in the new multi-access edge computing (MEC) architecture. Recently, several approaches have been proposed to solve the demand response problem for energy saving in cloud computing and MEC. However, accurate location information of edge web users should always be provided, which sacrifices users' privacy. To protect edge web users' location privacy while saving energy in MEC, we systematically formulate this location privacy-preserving edge demand response (LEDR) problem. To solve the LEDR problem effectively and efficiently, we propose a system named GEES by incorporating differential geo-obfuscation to secure user privacy while maximizing system utility and energy efficiency through inferences with theoretical analysis. Extensive and comprehensive experiments are conducted based on a synthetic real-world dataset, and the results demonstrate that GEES outperforms representative approaches by 23.02%, 31.47%, and 17.29% on average in terms of energy efficiency, user privacy and system utility.

The task of cardinality counting, pivotal for data analysis, endeavors to quantify unique elements within datasets and has significant applications across various sectors like healthcare, marketing, cybersecurity, and web analytics. Current methods, categorized into deterministic and probabilistic, often fail to prioritize data privacy. Given the fragmentation of datasets across various organizations, there is an elevated risk of inadvertently disclosing sensitive information during collaborative data studies using state-of-the-art cardinality counting techniques. This study introduces an innovative privacy-centric solution for the cardinality counting dilemma, leveraging a federated learning framework. Our approach involves employing a locally differentially private data encoding for initial processing, followed by a privacy-aware federated K-means clustering strategy, ensuring that cardinality counting occurs across distinct datasets without necessitating data amalgamation. The efficacy of our methodology is underscored by promising results from tests on both real-world and simulated datasets, pointing towards a transformative approach to privacy-sensitive cardinality counting in contemporary data science.

The energy industry is undergoing significant transformations as it strives to achieve net-zero emissions and future-proof its infrastructure, where every participant in the power grid has the potential to both consume and produce energy resources. Federated learning -- which enables multiple participants to collaboratively train a model without aggregating the training data -- becomes a viable technology. However, the global model parameters that have to be shared for optimization are still susceptible to training data leakage. In this work, we propose confined gradient descent (CGD) that enhances the privacy of federated learning by eliminating the sharing of global model parameters. CGD exploits the fact that a gradient descent optimization can start with a set of discrete points and converges to another set in the neighborhood of the global minimum of the objective function. As such, each participant can independently initiate its own private global model~(referred to as the confined model ), and collaboratively learn it towards the optimum. The updates to their own models are worked out in a secure collaborative way during the training process.In such a manner, CGD retains the ability of learning from distributed data but greatly diminishes information sharing. Such a strategy also allows the proprietary confined models to adapt to the heterogeneity in federated learning, providing inherent benefits of fairness. We theoretically and empirically demonstrate that decentralized CGD øne provides a stronger differential privacy (DP) protection; \two is robust against the state-of-the-art poisoning privacy attacks; þree results in bounded fairness guarantee among participants; and \four provides high test accuracy (comparable with centralized learning) with a bounded convergence rate over four real-world datasets.

Deploying Machine Learning as a Service gives rise to model plagiarism, leading to copyright infringement. Ownership testing techniques are designed to identify model fingerprints for verifying plagiarism. However, previous works often rely on overfitting or robustness features as fingerprints, lacking theoretical guarantees and exhibiting under-performance on generalized models. In this paper, we propose a novel ownership testing method called VeriDIP, whichverifies aDNN model'sintellectualproperty. VeriDIP makes two major contributions. (1) It utilizes membership inference attacks to estimate the lower bound of privacy leakage, which reflects the fingerprint of a given model. The privacy leakage fingerprints highlight the unique patterns through which the models memorize sensitive training datasets. (2) We introduce a novel approach using less private samples to enhance the performance of ownership testing. Extensive experimental results confirm that VeriDIP is effective and efficient in validating the ownership of deep learning models trained on both image and tabular datasets. VeriDIP achieves comparable performance to state-of-the-art methods on image datasets while significantly reducing computation and communication costs. Enhanced VeriDIP demonstrates superior verification performance on generalized deep learning models, particularly on table-trained models. Additionally, VeriDIP exhibits similar effectiveness on utility-preserving differentially private models compared to non-differentially private baselines.

Due to their low latency and high privacy preservation, there is currently a burgeoning demand for deploying deep learning (DL) models on ubiquitous edge Internet of Things (IoT) devices. However, DL models are often large in size and require large-scale computation, which prevents them from being placed directly onto IoT devices, where resources are constrained, and 32-bit floating-point (float-32) operations are unavailable. Commercial framework (i.e., a set of toolkits) empowered model quantization is a pragmatic solution that enables DL deployment on mobile devices and embedded systems by effortlessly post-quantizing a large high-precision model (e.g., float-32) into a small low-precision model (e.g., int-8) while retaining the model inference accuracy. However, their usability might be threatened by security vulnerabilities. This work reveals that standard quantization toolkits can be abused to activate a backdoor. We demonstrate that a full-precision backdoored model which does not have any backdoor effect in the presence of a trigger—as the backdoor is dormant—can be activated by (i) TensorFlow-Lite (TFLite) quantization, the onlyproduct-readyquantization framework to date, and (ii) thebeta releasedPyTorch Mobile framework. In our experiments, we employ three popular model architectures (VGG16, ResNet18, and ResNet50), and train each across three popular datasets: MNIST, CIFAR10 and GTSRB. We ascertain that all trained float-32 backdoored models exhibit no backdoor effecteven in the presence of trigger inputs. Particularly, four influential backdoor defenses are evaluated, and they fail to identify a backdoor in the float-32 models. When each of the float-32 models is converted into an int-8 format model through the standard TFLite or PyTorch Mobile framework's post-training quantization, the backdoor is activated in the quantized model, which shows a stable attack success rate close to 100% upon inputs with the trigger, while it usually behaves upon non-trigger inputs. This work highlights that a stealthy security threat occurs when an end-user utilizes the on-device post-training model quantization frameworks, informing security researchers of a cross-platform overhaul of DL models post-quantization even if these models pass security-aware front-end backdoor inspections. Significantly, we have identified Gaussian noise injection into the malicious full-precision model as an easy-to-use preventative defense against the PQ backdoor. The attack source code is released athttps://github.com/quantization-backdoor.

AI-powered programming language generation (PLG) models have gained increasing attention due to their ability to generate source code of programs in a few seconds with a plain program description. Despite their remarkable performance, many concerns are raised over the potential risks of their development and deployment, such as legal issues of copyright infringement induced by training usage of licensed code, and malicious consequences due to the unregulated use of these models. In this paper, we present the first-of-its-kind study to systematically investigate the accountability of PLG models from the perspectives of both model development and deployment. In particular, we develop a holistic framework not only to audit the training data usage of PLG models, but also to identify neural code generated by PLG models as well as determine its attribution to a source model. To this end, we propose using membership inference to audit whether a code snippet used is in the PLG model's training data. In addition, we propose a learning-based method to distinguish between human-written code and neural code. In neural code attribution, through both empirical and theoretical analysis, we show that it is impossible to reliably attribute the generation of one code snippet to one model. We then propose two feasible alternative methods: one is to attribute one neural code snippet to one of the candidate PLG models, and the other is to verify whether a set of neural code snippets can be attributed to a given PLG model. The proposed framework thoroughly examines the accountability of PLG models which are verified by extensive experiments. The implementations of our proposed framework are also encapsulated into a new artifact, named CODEFORENSIC, to foster further research.

To mitigate recent insidious backdoor attacks on deep learning models, advances have been made by the research community. Nonetheless, state-of-the-art defenses are either limited to specific backdoor attacks (i.e., source-agnostic attacks) or non-user-friendly in that machine learning expertise and/or expensive computing resources are required. This work observes that all existing backdoor attacks have an inadvertent and inevitable intrinsic weakness, termed as non-transferability —that is, a trigger input hijacks a backdoored model but is not effective in another model that has not been implanted with the same backdoor. With this key observation, we propose non-transferability enabled backdoor detection to identify trigger inputs for a model-under-test during run-time. Specifically, our detection allows a potentially backdoored model-under-test to predict a label for an input. Moreover, our detection leverages a feature extractor to extract feature vectors for the input and a group of samples randomly picked from its predicted class label, and then compares the similarity between the input and the samples in the feature extractor’s latent space to determine whether the input is a trigger input or a benign one. The feature extractor can be provided by a reputable party or is a free pre-trained model privately reserved from any open platform (e.g., ModelZoo, GitHub, Kaggle) by a user and thus our detection does not require the user to have any machine learning expertise or perform costly computations. Extensive experimental evaluations on four common tasks affirm that our detection scheme has high effectiveness (low false acceptance rate) and usability (low false rejection rate) with low detection latency against different types of backdoor attacks.

In recent years, the development of machine learning-based solutions for software services, particularly for source code, has grown rapidly. It is witnessed that many machine learning models for software services require the input of source code snippets in a desired form of abstract syntax tree (AST), which is mostly generated from an external tool. However, such data pre-processing tasks could be done by different engineering tools, and the impact of these tools towards final models is often neglected. In this work, we aim to investigate the source code engineering impacts towards machine learning-based software services. Three different types of parsing tools are identified, which are parser generator, parsing library and parser developed for a certain purpose. They are thoroughly evaluated towards the impacts on the prediction model of Code2Vec for the prediction task of the method name in Java language. The collective result on the Java-small dataset shows that the generated ASTs differ a lot in terms of source code structures and contents when using different parsing tools. The difference could influence the performance of the trained model significantly. Our result suggests that when machine learning models are implemented for software services, especially for code-related tasks, the selection of parsing tools should be thoroughly considered during the data pre-processing stage. While there are some interesting findings on Java-med and Java-small, we anticipate this work could provide some insights for better ML-based software service solutions from the perspective of source code engineering.

Machine learning models have made significant breakthroughs across various domains. However, it is crucial to assess these models to obtain a complete understanding of their capabilities and limitations and ensure their effectiveness and reliability in solving real-world problems. In this paper, we present a framework, termed ML-Compass, that covers a broad range of machine learning abilities, including utility evaluation, neuron analysis, robustness evaluation, and interpretability examination. We use this framework to assess seven state-of-the-art classification models on four benchmark image datasets. Our results indicate that different models exhibit significant variation, even when trained on the same dataset. This highlights the importance of using the assessment framework to comprehend their behavior.

The rapid development of deep learning has demonstrated its potential for deployment in many intelligent service systems. However, some issues such as optimisation (e.g., how to reduce the deployment resources costs and further improve the detection speed), especially in scenarios where limited resources are available, remain challenging to address. In this paper, we aim to delve into the principles of deep neural networks, focusing on the importance of network neurons. The goal is to identify the neurons that exert minimal impact on model performances, thereby aiding in the process of model pruning. In this work, we have thoroughly considered the deep learning model pruning process with and without fine-tuning step, ensuring the model performance consistency. To achieve our objectives, we propose a methodology that employs adversarial attack methods to explore deep neural network parameters. This approach is combined with an innovative attribution algorithm to analyse the level of network neurons involvement. In our experiments, our approach can effectively quantify the importance of network neuron. We extend the evaluation through comprehensive experiments conducted on a range of datasets, including CIFAR-10, CIFAR-100 and Caltech101. The results demonstrate that, our method have consistently achieved the state-of-the-art performance over many existing methods. We anticipate that this work will help to reduce the heavy training and inference cost of deep neural network models where a lightweight deep learning enhanced service and system is possible. The source code is open source at https://github.com/LMBTough/FVW.

While graph neural networks (GNNs) dominate the state-of-the-art for exploring graphs in real-world applications, they have been shown to be vulnerable to a growing number of privacy attacks. For instance, link stealing is a well-known membership inference attack (MIA) on edges that infers the presence of an edge in a GNN’s training graph. Recent studies on independent and identically distributed data (e.g., images) have empirically demonstrated that individuals from different groups suffer from different levels of privacy risks to MIAs, i.e., uneven vulnerability. However, theoretical evidence of such uneven vulnerability is missing. In this paper, we first present theoretical evidence of the uneven vulnerability of GNNs to link stealing attacks, which lays the foundation for demystifying such uneven risks among different groups of edges. We further demonstrate a group-based attack paradigm to expose the practical privacy harm to GNN users derived from the uneven vulnerability of edges. Finally, we empirically validate the existence of obvious uneven vulnerability on nine real-world datasets (e.g., about 25% AUC difference between different groups in the Credit graph). Compared with existing methods, the outperformance of our group-based attack paradigm confirms that customising different strategies for different groups results in more effective privacy attacks.

Anti-spoofing detection has become a necessity for face recognition systems due to the security threat posed by spoofing attacks. Despite great success in traditional attacks, most deep-learning-based methods perform poorly in 3D masks, which can highly simulate real faces in appearance and structure, suffering generalizability insufficiency while focusing only on the spatial domain with single frame input. This has been mitigated by the recent introduction of a biomedical technology called rPPG (remote photoplethysmography). However, rPPG-based methods are sensitive to noisy interference and require at least one second (> 25 frames) of observation time, which induces high computational overhead. To address these challenges, we propose a novel 3D mask detection framework, called FASTEN (Flow-Attention-based Spatio-Temporal aggrEgation Network). We tailor the network for focusing more on fine-grained details in large movements, which can eliminate redundant spatio-temporal feature interference and quickly capture splicing traces of 3D masks in fewer frames. Our proposed network contains three key modules: 1) a facial optical flow network to obtain non-RGB inter-frame flow information; 2) flow attention to assign different significance to each frame; 3) spatio-temporal aggregation to aggregate high-level spatial features and temporal transition features. Through extensive experiments, FASTEN only requires five frames of input and outperforms eight competitors for both intra-dataset and cross-dataset evaluations in terms of multiple detection metrics. Moreover, FASTEN has been deployed in real-world mobile devices for practical 3D mask detection.

Numerous open-source and commercial malware detectors are available. However, their efficacy is threatened by new adversarial attacks, whereby malware attempts to evade detection, e.g., by performing feature-space manipulation. In this work, we propose an explainability-guided and model-agnostic testing framework for robustness of malware detectors when confronted with adversarial attacks. The framework introduces the concept of Accrued Malicious Magnitude (AMM) to identify which malware features could be manipulated to maximize the likelihood of evading detection. We then use this framework to test several state-of-the-art malware detectors' ability to detect manipulated malware. We find that (i) commercial antivirus engines are vulnerable to AMM-guided test cases; (ii) the ability of a manipulated malware generated using one detector to evade detection by another detector (i.e., transferability) depends on the overlap of features with large AMM values between the different detectors; and (iii) AMM values effectively measure the fragility of features (i.e., capability of feature-space manipulation to flip the prediction results) and explain the robustness of malware detectors facing evasion attacks. Our findings shed light on the limitations of current malware detectors, as well as how they can be improved.

Video classification systems are vulnerable to adversarial attacks, which can create severe security problems in video verification. Current black-box attacks need a large number of queries to succeed, resulting in high computational overhead in the process of attack. On the other hand, attacks with restricted perturbations are ineffective against defenses such as denoising or adversarial training. In this paper, we focus on unrestricted perturbations and propose StyleFool, a black-box video adversarial attack via style transfer to fool the video classification system. StyleFool first utilizes color theme proximity to select the best style image, which helps avoid unnatural details in the stylized videos. Meanwhile, the target class confidence is additionally considered in targeted attacks to influence the output distribution of the classifier by moving the stylized video closer to or even across the decision boundary. A gradient-free method is then employed to further optimize the adversarial perturbations. We carry out extensive experiments to evaluate StyleFool on two standard datasets, UCF-101 and HMDB-51. The experimental results demonstrate that StyleFool outperforms the state-of-the-art adversarial attacks in terms of both the number of queries and the robustness against existing defenses. Moreover, 50% of the stylized videos in untargeted attacks do not need any query since they can already fool the video classification model. Furthermore, we evaluate the indistinguishability through a user study to show that the adversarial samples of StyleFool look imperceptible to human eyes, despite unrestricted perturbations.

Existing integrity verification approaches for deep models are designed for private verification (i.e., assuming the service provider is honest, with white-box access to model parameters). However, private verification approaches do not allow model users to verify the model at run-time. Instead, they must trust the service provider, who may tamper with the verification results. In contrast, a public verification approach that considers the possibility of dishonest service providers can benefit a wider range of users. In this paper, we propose PublicCheck, a practical public integrity verification solution for services of run-time deep models. PublicCheck considers dishonest service providers, and overcomes public verification challenges of being lightweight, providing anti-counterfeiting protection, and having fingerprinting samples that appear smooth. To capture and fingerprint the inherent prediction behaviors of a run-time model, PublicCheck generates smoothly transformed and augmented encysted samples that are enclosed around the model's decision boundary while ensuring that the verification queries are indistinguishable from normal queries. PublicCheck is also applicable when knowledge of the target model is limited (e.g., with no knowledge of gradients or model parameters). A thorough evaluation of PublicCheck demonstrates the strong capability for model integrity breach detection (100% detection accuracy with less than 10 black-box API queries) against various model integrity attacks and model compression attacks. PublicCheck also demonstrates the smooth appearance, feasibility, and efficiency of generating a plethora of encysted samples for fingerprinting.

The digital age has brought a world of opportunity to children. Connectivity can be a game-changer for some of the world’s most marginalized children. However, while legislatures around the world have enacted regulations to protect children’s online privacy, and app stores have instituted various protections, privacy in mobile apps remains a growing concern for parents and wider society. In this paper, we explore the potential privacy issues and threats that exist in these apps. We investigate 20195 mobile apps from the Google Play store that are designed particularly for children (Family apps) or include children in their target user groups (Normal apps). Using both static and dynamic analysis, we find that 4.47% of Family apps request location permissions, even though collecting location information from children is forbidden by the Play store, and 81.25% of Family apps use trackers (which are not allowed in children’s apps). Even major developers with 40+ kids apps on the Play store use ad trackers. Furthermore, we find that most permission request notifications are not well designed for children, and 19.25% apps have inconsistent content age ratings across the different protection authorities. Our findings suggest that, despite significant attention to children’s privacy, a large gap between regulatory provisions, app store policies, and actual development practices exist. Our research sheds light for government policymakers, app stores, and developers.

The Poisoning Membership Inference Attack (PMIA) is a newly emerging privacy attack that poses a significant threat to federated learning (FL). An adversary conducts data poisoning (i.e., performing adversarial manipulations on training examples) to extract membership information by exploiting the changes in loss resulting from data poisoning. The PMIA significantly exacerbates the traditional poisoning attack that is primarily focused on model corruption. However, there has been a lack of a comprehensive systematic study that thoroughly investigates this topic. In this work, we conduct a benchmark evaluation to assess the performance of PMIA against the Byzantine-robust FL setting that is specifically designed to mitigate poisoning attacks. We find that all existing coordinate-wise averaging mechanisms fail to defend against the PMIA, while the detect-then-drop strategy was proven to be effective in most cases, implying that the poison injection is memorized and the poisonous effect rarely dissipates. Inspired by this observation, we propose AgrEvader, a PMIA that maximizes the adversarial impact on the victim samples while circumventing the detection by Byzantine-robust mechanisms. AgrEvader significantly outperforms existing PMIAs. For instance, AgrEvader achieved a high attack accuracy of between 72.78% (on CIFAR-10) to 97.80% (on Texas100), which is an average accuracy increase of 13.89% compared to the strongest PMIA reported in the literature. We evaluated AgrEvader on five datasets across different domains, against a comprehensive list of threat models, which included black-box, gray-box and white-box models for targeted and non-targeted scenarios. AgrEvader demonstrated consistent high accuracy across all settings tested. The code is available at: https://github.com/PrivSecML/AgrEvader.

The advancement of secure communication and identity verification fields has significantly increased through the use of deep learning techniques for data hiding. By embedding information into a noise-tolerant signal, such as audio, video, or images, digital watermarking and steganography techniques can be used to protect sensitive intellectual property (IP) and enable confidential communication, ensuring that the information embedded is only accessible to authorized parties. This survey provides an overview of recent developments in deep learning techniques deployed for data hiding, categorized systematically according to model architectures and noise injection methods. In addition, potential future research directions that unite digital watermarking and steganography on software engineering to enhance security and mitigate risks are suggested and deliberated. This contribution furthers the creation of a more trustworthy digital world and advances responsible artificial intelligence (AI).

In this paper, we propose a novel and practical mechanism to enable the service provider to verify whether a suspect model is stolen from the victim model via model extraction attacks. Our key insight is that the profile of a DNN model's decision boundary can be uniquely characterized by its Universal Adversarial Perturbations (UAPs). UAPs belong to a low-dimensional subspace and piracy models' subspaces are more consistent with victim model's subspace compared with non-piracy model. Based on this, we propose a UAP fingerprinting method for DNN models and train an encoder via contrastive learning that takes fingerprints as inputs, outputs a similarity score. Extensive studies show that our framework can detect model Intellectual Property (IP) breaches with confidence > 99.99 % within only 20 fingerprints of the suspect model. It also has good generalizability across different model architectures and is robust against post-modifications on stolen models.

Coverage-guided Greybox Fuzzing (CGF) is one of the most successful and widely-used techniques for bug hunting. Two major approaches are adopted to optimize CGF: (i) to reduce search space of inputs by inferring relationships between input bytes and path constraints; (ii) to formulate fuzzing processes (e.g., path transitions) and build up probability distributions to optimize power schedules, i.e., the number of inputs generated per seed. However, the former is subjective to the inference results which may include extra bytes for a path constraint, thereby limiting the efficiency of path constraints resolution, code coverage discovery, and bugs exposure; the latter formalization, concentrating on power schedules for seeds alone, is inattentive to the schedule for bytes in a seed.

With the development of machine learning techniques, the attention of research has been moved from single-modal learning to multi-modal learning, as real-world data exist in the form of different modalities. However, multi-modal models often carry more information than single-modal models and they are usually applied in sensitive scenarios, such as medical report generation or disease identification. Compared with the existing membership inference against machine learning classifiers, we focus on the problem that the input and output of the multi-modal models are in different modalities, such as image captioning. This work studies the privacy leakage of multi-modal models through the lens of membership inference attack, a process of determining whether a data record involves in the model training process or not. To achieve this, we propose Multi-modal Models Membership Inference (M$^4$I) with two attack methods to infer the membership status, named metric-based (MB) M$^4$I and feature-based (FB) M$^4$I, respectively. More specifically, MB M$^4$I adopts similarity metrics while attacking to infer target data membership. FB M$^4$I uses a pre-trained shadow multi-modal feature extractor to achieve the purpose of data inference attack by comparing the similarities from extracted input and output features. Extensive experimental results show that both attack methods can achieve strong performances. Respectively, 72.5% and 94.83% of attack success rates on average can be obtained under unrestricted scenarios. Moreover, we evaluate multiple defense mechanisms against our attacks. The source code of M$^4$I attacks is publicly available at https://github.com/MultimodalMI/Multimodal-membership-inference.git.

The Android system manages access to sensitive APIs by permission enforcement. An application (app) must declare proper permissions before invoking specific Android APIs. However, there is no official documentation providing the complete list of permission-protected APIs and the corresponding permissions to date. Researchers have spent significant efforts extracting such API protection mapping from the Android API framework, which leverages static code analysis to determine if specific permissions are required before accessing an API. Nevertheless, none of them has attempted to analyze the protection mapping in the native library (i.e., code written in C and C++), an essential component of the Android framework that handles communication with the lower-level hardware, such as cameras and sensors. While the protection mapping can be utilized to detect various security vulnerabilities in Android apps, such as permission over-privilege, imprecise mapping will lead to false results in detecting such security vulnerabilities. To fill this gap, we thereby propose to construct the protection mapping involved in the native libraries of the Android framework to present a complete and accurate specification of Android API protection. We develop a prototype system, named NatiDroid, to facilitate the cross-language static analysis and compare its performance with two state-of-the-practice tools, termed Axplorer and Arcade. We evaluate NatiDroid on more than 11,000 Android apps, including system apps from custom Android ROMs and third-party apps from the Google Play. Our NatiDroid can identify up to 464 new API-permission mappings, in contrast to the worst-case results derived from both Axplorer and Arcade, where approximately 71% apps have at least one false positive in permission over-privilege. We have disclosed all the potential vulnerabilities detected to the stakeholders.

Deep neural networks (DNNs), regardless of their impressive performance, are vulnerable to attacks from adversarial inputs and, more recently, Trojans to misguide or hijack the decision of the model. We expose the existence of an intriguing class of spatially bounded, physically realizable, adversarial examples— Universal NaTuralistic adversarial paTches—we call TnTs, by exploring the super set of the spatially bounded adversarial example space and the natural input space within generative adversarial networks. Now, an adversary can arm themselves with a patch that is naturalistic, less malicious-looking, physically realizable, highly effective—achieving high attack success rates, and universal. A TnT is universal because any input image captured with a TnT in the scene will: i) misguide a network (untargeted attack); or ii) force the network to make a malicious decision (targeted attack). Interestingly, now, an adversarial patch attacker has the potential to exert a greater level of control—the ability to choose a location independent, natural-looking patch as a trigger in contrast to being constrained to noisy perturbations—an ability is thus far shown to be only possible with Trojan attack methods needing to interfere with the model building processes to embed a backdoor at the risk discovery; but, still realize a patch deployable in the physical world. Through extensive experiments on the large-scale visual classification task,ImageNetwith evaluations across its entire validation set of 50,000 images, we demonstrate the realistic threat from TnTs and the robustness of the attack. We show a generalization of the attack to create patches achieving higher attack success rates than existing state-of-the-art methods. Our results show the generalizability of the attack to different visual classification tasks (CIFAR-10,GTSRB,PubFig) and multiple state-of-the-art deep neural networks such as WideResnet50, Inception-V3 and VGG-16.

The ability to read, reason, and infer lies at the heart of neural reasoning architectures. After all, the ability to perform logical reasoning over language remains a coveted goal of Artificial Intelligence. To this end, models such as the Turing-complete differentiable neural computer (DNC) boast of real logical reasoning capabilities, along with the ability to reason beyond simple surface-level matching. In this brief, we propose the first probe into DNC's logical reasoning capabilities with a focus on text-based question answering (QA). More concretely, we propose a conceptually simple but effective adversarial attack based on metamorphic relations. Our proposed adversarial attack reduces DNCs' state-of-the-art accuracy from 100% to 1.5% in the worst case, exposing weaknesses and susceptibilities in modern neural reasoning architectures. We further empirically explore possibilities to defend against such attacks and demonstrate the utility of our adversarial framework as a simple scalable method to improve model adversarial robustness.

The proliferation of Internet of Things (IoT) devices has made people's lives more convenient, but it has also raised many security concerns. Due to the difficulty of obtaining and emulating IoT firmware, in the absence of internal execution information, black-box fuzzing of IoT devices has become a viable option. However, existing black-box fuzzers cannot form effective mutation optimization mechanisms to guide their testing processes, mainly due to the lack of feedback. In addition, because of the prevalent use of various and non-standard communication message formats in IoT devices, it is difficult or even impossible to apply existing grammar-based fuzzing strategies. Therefore, an efficient fuzzing approach with syntax inference is required in the IoT fuzzing domain.

Generative Adversarial Networks (GAN)-synthesized table publishing lets people privately learn insights without access to the private table. However, existing studies on Membership Inference (MI) Attacks show promising results on disclosing membership of training datasets of GAN-synthesized tables. Different from those works focusing on discovering membership of a given data point, in this paper, we propose a novel Membership Collision Attack against GANs (TableGAN-MCA), which allows an adversary given only synthetic entries randomly sampled from a black-box generator to recover partial GAN training data. Namely, a GAN-synthesized table immune to state-of-the-art MI attacks is vulnerable to the TableGAN-MCA. The success of TableGAN-MCA is boosted by an observation that GAN-synthesized tables potentially collide with the training data of the generator.

Natural language processing (NLP) systems have been proven to be vulnerable to backdoor attacks, whereby hidden features (backdoors) are trained into a language model and may only be activated by specific inputs (called triggers), to trick the model into producing unexpected behaviors. In this paper, we create covert and natural triggers for textual backdoor attacks, hidden backdoors, where triggers can fool both modern language models and human inspection. We deploy our hidden backdoors through two state-of-the-art trigger embedding methods. The first approach via homograph replacement, embeds the trigger into deep neural networks through the visual spoofing of lookalike characters replacement. The second approach uses subtle differences between text generated by language models and real natural text to produce trigger sentences with correct grammar and high fluency. We demonstrate that the proposed hidden backdoors can be effective across three downstream security-critical NLP tasks, representative of modern human-centric NLP systems, including toxic comment detection, neural machine translation (NMT), and question answering (QA). Our two hidden backdoor attacks can achieve an Attack Success Rate (ASR) of at least 97% with an injection rate of only 3% in toxic comment detection, 95.1% ASR in NMT with less than 0.5% injected data, and finally 91.12% ASR against QA updated with only 27 poisoning data samples on a model previously trained with 92,024 samples (0.029%). We are able to demonstrate the adversary's high success rate of attacks, while maintaining functionality for regular users, with triggers inconspicuous by the human administrators.

Along with gaining popularity of Real-Time Bidding (RTB) based programmatic advertising, the click farm based invalid traffic, which leverages massive real smartphones to carry out large-scale ad fraud campaigns, is becoming one of the major threats against online advertisement. In this study, we take an initial step towards the detection and large-scale measurement of the click farm based invalid traffic. Our study begins with a measurement on the device's features using a real-world labeled dataset, which reveals a series of features distinguishing the fraudulent devices from the benign ones. Based on these features, we develop EvilHunter, a system for detecting fraudulent devices through ad bid request logs with a focus on clustering fraudulent devices. EvilHunter functions by 1) building a classifier to distinguish fraudulent and benign devices; 2) clustering devices based on app usage patterns; and 3) relabeling devices in clusters through majority voting. EvilHunter demonstrates 97% precision and 95% recall on a real-world labeled dataset. By investigating a super click farm, we reveal several cheating strategies that are commonly adopted by fraudulent clusters. We further reduce the overhead of EvilHunter and discuss how to deploy the optimized EvilHunter in a real-world system. We are in partnership with a leading ad verification company to integrate EvilHunter into their industrial platform.

Although the use of pay-per-click mechanisms stimulates the prosperity of the mobile advertisement network, fraudulent ad clicks result in huge financial losses for advertisers. Extensive studies identify click fraud according to click/traffic patterns based on dynamic analysis. However, in this study, we identify a novel click fraud, named humanoid attack, which can circumvent existing detection schemes by generating fraudulent clicks with similar patterns to normal clicks. We implement the first tool ClickScanner to detect humanoid attacks on Android apps based on static analysis and variational AutoEncoders (VAEs) with limited knowledge of fraudulent examples. We define novel features to characterize the patterns of humanoid attacks in the apps' bytecode level. ClickScanner builds a data dependency graph (DDG) based on static analysis to extract these key features and form a feature vector. We then propose a classification model only trained on benign datasets to overcome the limited knowledge of humanoid attacks.

The rapid spread of COVID-19 has made manual contact tracing difficult. Thus, various public health authorities have experimented with automatic contact tracing using mobile applications (or "apps"). These apps, however, have raised security and privacy concerns. In this paper, we propose an automated security and privacy assessment tool - COVIDGUARDIAN - which combines identification and analysis of Personal Identification Information (PII), static program analysis and data flow analysis, to determine security and privacy weaknesses. Furthermore, in light of our findings, we undertake a user study to investigate concerns regarding contact tracing apps. We hope that COVIDGUARDIAN, and the issues raised through responsible disclosure to vendors, can contribute to the safe deployment of mobile contact tracing. As part of this, we offer concrete guidelines, and highlight gaps between user requirements and app performance.

Abstract With the rapid development of deep learning techniques, the popularity of voice services implemented on various Internet of Things (IoT) devices is ever increasing. In this paper, we examine user-level membership inference in the problem space of voice services, by designing an audio auditor to verify whether a specific user had unwillingly contributed audio used to train an automatic speech recognition (ASR) model under strict black-box access. With user representation of the input audio data and their corresponding translated text, our trained auditor is effective in user-level audit. We also observe that the auditor trained on specific data can be generalized well regardless of the ASR model architecture. We validate the auditor on ASR models trained with LSTM, RNNs, and GRU algorithms on two state-of-the-art pipelines, the hybrid ASR system and the end-to-end ASR system. Finally, we conduct a real-world trial of our auditor on iPhone Siri, achieving an overall accuracy exceeding 80%. We hope the methodology developed in this paper and findings can inform privacy advocates to overhaul IoT privacy.

Mobile phishing attacks, such as mimic mobile browser pages, masquerade as legitimate applications by leveraging repackaging or clone techniques, have caused varied yet significant security concerns. Consequently, detection techniques have been receiving increasing attention. However, many such detection methods are not well tested and may therefore still be vulnerable to new types of phishing attacks. In this article, we propose a new attacking technique, named GUI-Squatting attack, which can generate phishing apps (phapps) automatically and effectively on the Android platform. Our method adopts image processing and deep learning algorithms, to enable powerful and large-scale attacks. We observe that a successful phishing attack requires two conditions, page confusion and logic deception during attacks synthesis. We directly optimize these two conditions to create a practical attack. Our experimental results reveal that existing phishing defenses are less effective against such emergent attacks and may, therefore, stimulate more efficient detection techniques. To further demonstrate that our generatedphappscan not only bypass existing detection techniques, but also deceive real users, we conduct a human study and successfully steal users’ login information. The human study also shows that different response messages (e.g., “Crash” and “Server failed”) after pressing the login button mislead users to regard our phapps as functionality problems instead of security threats. Extensive experiments reveal that such newly proposed attacks still remain mostly undetected, and are worth further exploration.

Deep neural networks (DNNs) have been proven vulnerable to backdoor attacks, where hidden features (patterns) trained to a normal model, which is only activated by some specific input (called triggers), trick the model into producing unexpected behavior. In this article, we create covert and scattered triggers for backdoor attacks, invisible backdoors, where triggers can fool both DNN models and human inspection. We apply our invisible backdoors through two state-of-the-art methods of embedding triggers for backdoor attacks. The first approach on Badnets embeds the trigger into DNNs through steganography. The second approach of a trojan attack uses two types of additional regularization terms to generate the triggers with irregular shape and size. We use the Attack Success Rate and Functionality to measure the performance of our attacks. We introduce two novel definitions of invisibility for human perception; one is conceptualized by the Perceptual Adversarial Similarity Score (PASS) and the other is Learned Perceptual Image Patch Similarity (LPIPS). We show that the proposed invisible backdoors can be fairly effective across various DNN models as well as four datasets MNIST, CIFAR-10, CIFAR-100, and GTSRB, by measuring their attack success rates for the adversary, functionality for the normal users, and invisibility scores for the administrators. We finally argue that the proposed invisible backdoor attacks can effectively thwart the state-of-the-art trojan backdoor detection approaches.

With the rise of third parties in the machine learning pipeline, the service provider in “Machine Learning as a Service” (MLaaS), or external data contributors in online learning, or the retraining of existing models, the need to ensure the security of the resulting machine learning models has become an increasingly important topic. The security community has demonstrated that without transparency of the data and the resulting model, there exist many potential security risks, with new risks constantly being discovered. In this paper, we focus on one of these security risks - poisoning attacks. Specifically, we analyze how attackers may interfere with the results of regression learning by poisoning the training datasets. To this end, we analyze and develop a new poisoning attack algorithm. Our attack, termed Nopt, in contrast with previous poisoning attack algorithms, can produce larger errors with the same proportion of poisoning data-points. Furthermore, we also significantly improve the state-of-the-art defense algorithm, termed TRIM, proposed by Jagielsk et al. (IEEE S&P 2018), by incorporating the concept of probability estimation of clean data-points into the algorithm. Our new defense algorithm, termed Proda, demonstrates an increased effectiveness in reducing errors arising from the poisoning dataset through optimizing ensemble models. We highlight that the time complexity of TRIM had not been estimated; however, we deduce from their work that TRIM can take exponential time complexity in the worst-case scenario, in excess of Proda's logarithmic time. The performance of both our proposed attack and defense algorithms is extensively evaluated on four real-world datasets of housing prices, loans, health care, and bike sharing services. We hope that our work will inspire future research to develop more robust learning algorithms immune to poisoning attacks.

Online Automated Privacy Policy Generators (APPGs) are tools used by app developers to quickly create app privacy policies which are required by privacy regulations to be incorporated to each mobile app. The creation of these tools brings convenience to app developers; however, the quality of these tools puts developers and stakeholders at legal risk. In this paper, we conduct an empirical study to assess the quality of online APPGs. We analyze the completeness of privacy policies, determine what categories and items should be covered in a complete privacy policy, and conduct APPG assessment with boilerplate apps. The results of assessment show that due to the lack of static or dynamic analysis of app's behavior, developers may encounter two types of issues caused by APPGs. First, the generated policies could be incomplete because they do not cover all the essential items required by a privacy policy. Second, some generated privacy policies contain unnecessary personal information collection or arbitrary commitments inconsistent with user input. Ultimately, the defects of APPGs may potentially lead to serious legal issues. We hope that the results and insights developed in this paper can motivate the healthy and ethical development of APPGs towards generating a more complete, accurate, and robust privacy policy.

Mobile banking apps, belonging to the most security-critical app category, render massive and dynamic transactions susceptible to security risks. Given huge potential financial loss caused by vulnerabilities, existing research lacks a comprehensive empirical study on the security risks of global banking apps to provide useful insights and improve the security of banking apps.

Since the new privacy feature in iOS enabling users to acknowledge which app is reading or writing to his or her clipboard through prompting notifications was updated, a plethora of top apps have been reported to frequently access the clipboard without user consent. However, the lack of monitoring and control of Android application's access to the clipboard data leave Android users blind to their potential to leak private information from Android clipboards, raising severe security and privacy concerns. In this preliminary work, we envisage and investigate an approach to (i) dynamically detect clipboard access behaviour, and (ii) determine privacy leaks via static data flow analysis, in which we enhance the results of taint analysis with call graph concatenation to enable leakage source backtracking. Our preliminary results indicate that the proposed method can expose clipboard data leakage as substantiated by our discovery of a popular app, i.e., Sogou Input, directly monitoring and transferring user data in a clipboard to backend servers.

Rapid spread of the COVID-19 pandemic is making traditional manual contact tracing challenging; in response, digital contact tracing mobile apps have been developed by the software industry and promoted by governments and health authorities worldwide. However, deploying contact tracing apps across a population at scale have raised many privacy concerns. In this paper, we propose a venue-access-based contact tracing solution, VenueTrace, which preserves user privacy by designs by: (i) enabling the contact tracing of venue-to-user, instead of user-to-user; (ii) avoiding information exchanges between users; and (iii) ensuring no private data is exposed to back-end servers, while enabling proximity contact tracing.

The past decade has seen the great potential of applying deep neural network (DNN) based software to safety-critical scenarios, such as autonomous driving. Similar to traditional software, DNNs could exhibit incorrect behaviors, caused by hidden defects, leading to severe accidents and losses. In this paper, we propose DeepHunter, a coverage-guided fuzz testing framework for detecting potential defects of general-purpose DNNs. To this end, we first propose a metamorphic mutation strategy to generate new semantically preserved tests, and leverage multiple extensible coverage criteria as feedback to guide the test generation. We further propose a seed selection strategy that combines both diversity-based and recency-based seed selection. We implement and incorporate 5 existing testing criteria and 4 seed selection strategies in DeepHunter. Large-scale experiments demonstrate that (1) our metamorphic mutation strategy is useful to generate new valid tests with the same semantics as the original seed, by up to a 98% validity ratio; (2) the diversity-based seed selection generally weighs more than recency-based seed selection in boosting the coverage and in detecting defects; (3) DeepHunter outperforms the state of the arts by coverage as well as the quantity and diversity of defects identified; (4) guided by corner-region based criteria, DeepHunter is useful to capture defects during the DNN quantization for platform migration.

Last-mile geo-localization plays an essential role in many location-based services, such as fraud detection and targeted advertising. In this study, we point out that round trip time (RTT) latency shows an extremely weak correlation with physical distance estimation in China's Internet, since a path between a vantage point and a destination can often be circuitous and inflated by queuing and processing delays. To sidestep the latency measurement, we perform a three-tier hop count based IP geo-localization mapping for China's Internet, on the assumption that each provincial router only serves a limited area. The mapping approach begins at the first tier using a single vantage point to fetch large-scale traceroute paths from the server to landmarks and target IPs. At the second tier, we try to find the last common routers along the traceroute paths of targets and landmarks and aggregate their hop count distances. At the third tier, we estimate the physical distances from hop count distances and provincial router radii, and geo-localize the targets to the nearest landmarks. Through large-scale experiments, we show that our approach is both cost-efficient and reliable, and can achieve last-ten-kilometer IP geo-localization for approximately 65% of the total 48874 pingable target IP addresses with a single ping server, and our hop count based approach completely outperforms the RTT based method.

Misspelled keywords have become an appealing target in search poisoning, since they are less competitive to promote than the correct queries and account for a considerable amount of search traffic. Search engines have adopted several countermeasure strategies, e.g., Google applies automated corrections on queried keywords and returns search results of the corrected versions directly. However, a sophisticated class of attack, which we term as linguistic-collision misspelling, can evade auto-correction and poison search results. Cybercriminals target special queries where the misspelled terms are existent words, even in other languages (e.g., "idobe", a misspelling of the English word "adobe", is a legitimate word in the Nigerian language). In this paper, we perform the first large-scale analysis on linguistic-collision search poisoning attacks. In particular, we check 1.77 million misspelled search terms on Google and Baidu and analyze both English and Chinese languages, which are the top two languages used by Internet users. We leverage edit distance operations and linguistic properties to generate misspelling candidates. To more efficiently identify linguistic-collision search terms, we design a deep learning model that can improve collection rate by 2.84x compared to random sampling. Our results show that the abuse is prevalent: around 1.19% of linguistic-collision search terms on Google and Baidu have results on the first page directing to malicious websites. We also find that cybercriminals mainly target categories of gambling, drugs, and adult content. Mobile-device users disproportionately search for misspelled keywords, presumably due to small screen for input. Our work highlights this new class of search engine poisoning and provides insights to help mitigate the threat.

Deep learning (DL) has achieved remarkable progress over the past decade and has been widely applied to many industry domains. However, the robustness of DL systems recently becomes great concerns, where minor perturbation on the input might cause the DL malfunction. These robustness issues could potentially result in severe consequences when a DL system is deployed to safety-critical applications and hinder the real-world deployment of DL systems. Testing techniques enable the robustness evaluation and vulnerable issue detection of a DL system at an early stage. The main challenge of testing a DL system attributes to the high dimensionality of its inputs and large internal latent feature space, which makes testing each state almost impossible. For traditional software, combinatorial testing (CT) is an effective testing technique to balance the testing exploration effort and defect detection capabilities. In this paper, we perform an exploratory study of CT on DL systems. We propose a set of combinatorial testing criteria specialized for DL systems, as well as a CT coverage guided test generation technique. Our evaluation demonstrates that CT provides a promising avenue for testing DL systems.

As smart phones gradually become the dominant network traffic generators, app traffic analysis methods have gained great interests for network management and targeted advertisement. Specifically, previous works have shown that the scalability of app inference via traffic meta-data has the edge over traditional payload based analysis. However, such works mainly considered the ideal inference scenario where only one app is running on the client's device, without any background traffic noise interfered. In this paper, we extend the research to a more practical scenario, by assuming that multiple apps simultaneously run on a smart phone in the noisy background with complex traffic generated by the operating system. To that end, we propose APPCLASSIFIER, an Android app fingerprinting scheme for real-time app inference. We first leverage the observed differences in packet size distributions and traffic sequential behaviors to boost inference accuracy for noise-free traffic analysis. We then propose novel heuristic based methods to re-correct mislabeled traffic flows to realize real-time traffic inference. As a result, APPCLASSIFIER achieves inference accuracy of 82.3% for noise-free traffic analysis and reduces error rate from 66.7% to 36.4% for real-time traffic inference.

Deep learning (DL) defines a new data-driven programming paradigm where the internal system logic is largely shaped by the training data. The standard way of evaluating DL models is to examine their performance on a test dataset. The quality of the test dataset is of great importance to gain confidence of the trained models. Using an inadequate test dataset, DL models that have achieved high test accuracy may still lack generality and robustness. In traditional software testing, mutation testing is a well-established technique for quality evaluation of test suites, which analyzes to what extent a test suite detects the injected faults. However, due to the fundamental difference between traditional software and deep learning-based software, traditional mutation testing techniques cannot be directly applied to DL systems. In this paper, we propose a mutation testing framework specialized for DL systems to measure the quality of test data. To do this, by sharing the same spirit of mutation testing in traditional software, we first define a set of source-level mutation operators to inject faults to the source of DL (i.e., training data and training programs). Then we design a set of model-level mutation operators that directly inject faults into DL models without a training process. Eventually, the quality of test data could be evaluated from the analysis on to what extent the injected faults could be detected. The usefulness of the proposed mutation testing techniques is demonstrated on two public datasets, namely MNIST and CIFAR-10, with three DL models.

Deep learning (DL) defines a new data-driven programming paradigm that constructs the internal system logic of a crafted neuron network through a set of training data. We have seen wide adoption of DL in many safety-critical scenarios. However, a plethora of studies have shown that the state-of-the-art DL systems suffer from various vulnerabilities which can lead to severe consequences when applied to real-world applications. Currently, the testing adequacy of a DL system is usually measured by the accuracy of test data. Considering the limitation of accessible high quality test data, good accuracy performance on test data can hardly provide confidence to the testing adequacy and generality of DL systems. Unlike traditional software systems that have clear and controllable logic and functionality, the lack of interpretability in a DL system makes system analysis and defect detection difficult, which could potentially hinder its real-world deployment. In this paper, we propose DeepGauge, a set of multi-granularity testing criteria for DL systems, which aims at rendering a multi-faceted portrayal of the testbed. The in-depth evaluation of our proposed testing criteria is demonstrated on two well-known datasets, five DL systems, and with four state-of-the-art adversarial attack techniques against DL. The potential usefulness of DeepGauge sheds light on the construction of more generic and robust DL systems.

Mobile banking apps, as one of the most contemporary FinTechs, have been widely adopted by banking entities to provide instant financial services. However, our recent work discovered thousands of vulnerabilities in 693 banking apps, which indicates these apps are not as secure as we expected. This motivates us to conduct this study for understanding the current security status of them. First, we take 6 months to track the reporting and patching procedure of these vulnerabilities. Second, we audit 4 state-of the-art vulnerability detection tools on those patched vulnerabilities. Third, we discuss with 7 banking entities via in-person or online meetings and conduct an online survey to gain more feedback from financial app developers. Through this study, we reveal that (1) people may have inconsistent understandings of the vulnerabilities and different criteria for rating severity; (2) state-of-the-art tools are not effective in detecting vulnerabilities that the banking entities most concern; and (3) more efforts should be endeavored in different aspects to secure banking apps. We believe our study can help bridge the existing gaps, and further motivate different parties, including banking entities, researchers and policy makers, to better tackle security issues altogether.

FinTech, short for ``financial technology,'' has advanced the process of transforming financial business from a traditional manual-process-driven to an automation-driven model by providing various software platforms. However, the current FinTech-industry still heavily depends on manual testing, which becomes the bottleneck of FinTech industry development. To automate the testing process, we propose an approach of black-box testing for a FinTech system with effective tool support for both test generation and test oracles. For test generation, we first extract input categories from business-logic specifications, and then mutate real data collected from system logs with values randomly picked from each extracted input category. For test oracles, we propose a new technique of priority differential testing where we evaluate execution results of system-test inputs on the system's head (i.e., latest) version in the version repository (1) against the last legacy version in the version repository (only when the executed test inputs are on new, not-yet-deployed services) and (2) against both the currently-deployed version and the last legacy version (only when the test inputs are on existing, deployed services). When we rank the behavior-inconsistency results for developers to inspect, for the latter case, we give the currently-deployed version as a higher-priority source of behavior to check. We apply our approach to the CSTP subsystem, one of the largest data processing and forwarding modules of the China Foreign Exchange Trade System (CFETS) platform, whose annual total transaction volume reaches 150 trillion US dollars. Extensive experimental results show that our approach can substantially boost the branch coverage by approximately 40%, and is also efficient to identify common faults in the FinTech system.

The evolution of mobile malware poses a serious threat to smartphone security. Today, sophisticated attackers can adapt by maximally sabotaging machine-learning classifiers via polluting training data , rendering most recent machine learning-based malware detection tools (such as D rebin , D roid APIM iner , and M a M a D roid ) ineffective. In this paper, we explore the feasibility of constructing crafted malware samples ; examine how machine-learning classifiers can be misled under three different threat models; then conclude that injecting carefully crafted data into training data can significantly reduce detection accuracy. To tackle the problem, we propose K uafu D et , a two-phase learning enhancing approach that learns mobile malware by adversarial detection. K uafu D et includes an offline training phase that selects and extracts features from the training set, and an online detection phase that utilizes the classifier trained by the first phase. To further address the adversarial environment, these two phases are intertwined through a self-adaptive learning scheme, wherein an automated camouflage detector is introduced to filter the suspicious false negatives and feed them back into the training phase. We finally show that K uafu D et can significantly reduce false negatives and boost the detection accuracy by at least 15%. Experiments on more than 250,000 mobile applications demonstrate that K uafu D et is scalable and can be highly effective as a standalone system.

Despite recent progress in program analysis techniques to identify vulnerabilities in Android apps, significant challenges still remain for applying these techniques to large-scale industrial environments. Modern software-security providers, such as Qihoo 360 and Pwnzen (two leading companies in China), are often required to process more than 10 million mobile apps at each run. In this work, we focus on effectively and efficiently identifying vulnerable usage of Internet sockets in an industrial setting. To achieve this goal, we propose a practical hybrid approach that enables lightweight yet precise detection in the industrial setting. In particular, we integrate the process of categorizing potential vulnerable apps with analysis techniques, to reduce the inevitable human inspection effort. We categorize potential vulnerable apps based on characteristics of vulnerability signatures, to reduce the burden on static analysis. We flexibly integrate static and dynamic analyses for apps in each identified family, to refine the family signatures and hence target on precise detection. We implement our approach in a practical system and deploy the system on the Pwnzen platform. By using the system, we identify and report potential vulnerabilities of 24 vulnerable apps (falling into 3 vulnerability families) to their developers, and some of these reported vulnerabilities are previously unknown. The apps of each vulnerability family in total have over 50 million downloads. We also propose countermeasures and highlight promising directions for technology transfer.

Mobile devices are especially vulnerable nowadays to malware attacks, thanks to the current trend of increased app downloads. Despite the significant security and privacy concerns it received, effective malware detection (MD) remains a significant challenge. This paper tackles this challenge by introducing a streaminglized machine learning-based MD framework, StormDroid: (i) The core of StormDroid is based on machine learning, enhanced with a novel combination of contributed features that we observed over a fairly large collection of data set; and (ii) we streaminglize the whole MD process to support large-scale analysis, yielding an efficient and scalable MD technique that observes app behaviors statically and dynamically. Evaluated on roughly 8,000 applications, our combination of contributed features improves MD accuracy by almost 10% compared with state-of-the-art antivirus systems; in parallel our streaminglized process, StormDroid, further improves efficiency rate by approximately three times than a single thread.

This paper proposes Begonia, a malware detection system through Pareto ensemble pruning. We convert the malware detection problem into the bi-objective Pareto optimization, aiming to trade off the classification accuracy and the size of classifiers as two objectives. We automatically generate several groups of base classifiers using SVM and generate solutions through bi-objective Pareto optimization. We then select the ensembles with highest accuracy of each group to form the final solutions, among which we hit the optimal solution where the combined loss function is minimal considering the trade-off between accuracy and time cost. We expect users to provide different trade-off levels to their different requirements to select the best solution. Experimental results show that Begonia can achieve higher accuracy with relatively lower overhead compared to the ensemble containing all the classifiers and can make a good trade-off to different requirements.

Android malware has been found on various third-party online markets, which poses drastic threats to mobile users in terms of security and privacy. Machine learning is one of the promising approaches to discriminate the malicious applications from the benign ones. Despite its higher malware detection capability, a significant challenge remains: in adversarial environment, an attacker can adapt by maximally sabotaging classifiers by polluting training data. This paper proposes KuafuDet, a two-phase learning enhancing approach that adversarially detects the Android malware. Experiments on more than 50,000 Android applications demonstrate the effectiveness and scalability of our approach.

Abstract Due to the recent “Right to be Forgotten” (RTBF) ruling, for queries about an individual, Google and other search engines now delist links to web pages that contain “inadequate, irrelevant or no longer relevant, or excessive” information about that individual. In this paper we take a data-driven approach to study the RTBF in the traditional media outlets, its consequences, and its susceptibility to inference attacks. First, we do a content analysis on 283 known delisted UK media pages, using both manual investigation and Latent Dirichlet Allocation (LDA). We find that the strongest topic themes are violent crime, road accidents, drugs, murder, prostitution, financial misconduct, and sexual assault. Informed by this content analysis, we then show how a third party can discover delisted URLs along with the requesters’ names, thereby putting the efficacy of the RTBF for delisted media links in question. As a proof of concept, we perform an experiment that discovers two previously-unknown delisted URLs and their corresponding requesters. We also determine 80 requesters for the 283 known delisted media pages, and examine whether they suffer from the “Streisand effect,” a phenomenon whereby an attempt to hide a piece of information has the unintended consequence of publicizing the information more widely. To measure the presence (or lack of presence) of a Streisand effect, we develop novel metrics and methodology based on Google Trends and Twitter data. Finally, we carry out a demographic analysis of the 80 known requesters. We hope the results and observations in this paper can inform lawmakers as they refine RTBF laws in the future.

Abstract Location‐based social discovery (LBSD) services enable users to discover their geographic neighborhoods to make new friends. Original LBSD services were designed to provide the exact distances to nearby users. It has been shown that it is easy to pinpoint any target user's location by using trilateration based on the exact distances from three fake Global Positioning System locations to the target user. To defend against the trilateration attack, contemporary LBSD services then began to report distances of nearby users in concentric bands, for example, bands of 100 meters, rather than exact distances. In this paper, we investigate the user location privacy leakage problem in LBSD services reporting distances in discrete bands. Using number theory, we analytically show that by strategically placing multiple virtual probes with fake Global Positioning System locations, one can nevertheless localize user locations in band‐based LBSD. Our methodology is guaranteed to localize any reported user within a circle of radius no greater than one meter, even for LBSD services using large bands (such as 100 m as used by WeChat). Eventually, countermeasures are proposed to reduce location privacy leakage to the very minimum. To the best of our knowledge, this is the first work that explicitly exploits and quantifies user location privacy leakage in band‐based LBSD services. We expect our study to draw more public attention to this serious privacy issue and expectantly motivate better privacy preserving LBSD designs. Copyright © 2016 John Wiley & Sons, Ltd.