The Deep Neural Network (DNN) based detection model’s dependency on third-party crowdsourced sources poses a new security threat from backdoor attacks against malware detectors. Attackers attempt to inject hidden backdoors into the target model, allowing it to perform well on clean samples. Once the attacker-defined trigger activates the hidden backdoor, the model predictions for poisoned samples are maliciously altered. Different from existing backdoor attacks either utilize fixed triggers or generate sample-specific triggers, we explore a novel backdoor attack paradigm in malware domain and propose MalPDT, in which backdoor triggers achieve dynamic variability in trigger patterns and retain compatibility across malware samples. We train a generator capable of hiding information to produce dynamically variable encoded byte segments, which are then injected as triggers into the unused regions of PE malware in a functionality-preserving manner. In MalPDT, any combination of a malware sample and a trigger can form a poisoned sample capable of activating the backdoor, enabling plug-and-play capability. We conduct extensive experiments to validate the effectiveness of MalPDT in attacking models with or without defenses.

Generating adversarial examples (AEs) can be formulated as an optimization problem. Among various optimization-based attacks, the gradient-based PGD and the momentum-based MI-FGSM have garnered considerable interest. However, all these attacks use the sign function to scale their perturbations, which raises several theoretical concerns from the point of view of optimization. In this paper, we first reveal that PGD is actually a specific reformulation of the projected gradient method using only the current gradient to determine its step-size. Further, we show that when we utilize a conventional adaptive matrix with the accumulated gradients to scale the perturbation, PGD becomes AdaGrad. Motivated by this analysis, we present a novel momentum-based attack AdaMI, in which the perturbation is optimized with an interesting momentum-based adaptive matrix. AdaMI is proved to attain optimal convergence for convex problems, indicating that it addresses the non-convergence issue of MI-FGSM, thereby ensuring stability of the optimization process. The experiments demonstrate that the proposed momentum-based adaptive matrix can serve as a general and effective technique to boost adversarial transferability over the state-of-the-art methods across different networks while maintaining better stability and imperceptibility.

The adversarial example presents new security threats to trustworthy detection systems. In the context of evading dynamic detection based on API call sequences, a practical approach involves inserting perturbing API calls to modify these sequences. The type of inserted API calls and their insertion locations are crucial for generating an effective adversarial API call sequence. Existing methods either optimize the inserted API calls while neglecting the insertion positions or treat these optimizations as separate processes. This can lead to inefficient attacks that insert a large number of unnecessary API calls. To address this issue, we propose a novel reinforcement learning (RL) framework, dubbed GAME-RL, which simultaneously optimizes both the perturbing APIs and their insertion positions. Specifically, we define malware modification through IAT (Import Address Table) hooking as a sequential decision-making process. We introduce an invalid action masking and an auto-regressive policy head within the RL framework, ensuring the feasibility of IAT hooking and capturing the inherent relationship between factors. GAME-RL learns more effective evasion strategies, taking into account functionality preservation and the black-box setting. We conduct comprehensive experiments on various target models, demonstrating that GAME-RL significantly improves the evasion rate while maintaining acceptable levels of adversarial overhead.

With the widespread application of machine learning techniques in malware detection, researchers have proposed various adversarial attack methods to generate adversarial examples (AEs) of malware, thereby evading detection. Previous studies have shown that the reinforcement learning (RL) framework can enable black-box attacks by performing a sequence of function-preserving operations, which produces functional evasive malware samples. However, it is difficult to obtain the useful guidance and feedbacks from the environment for agent training in the black-box scenario, which results in the RL framework being unable to learn the effective evasion policy. In this paper, we propose the Shapley prior and establish a prior-guidance-based RL framework, namely PSP-Mal, to generate AEs against Portable Executable (PE) malware detectors. Our framework improves on existing methods in three aspects: 1) We explore feature effects of the black-box model by computing Shapley values and further propose the Shapley prior to represent the expected impact of operations. 2) A novel prioritized experience utilization mechanism is established regarding the Shapley prior guidance in the RL framework. 3) The actions are expanded into item-content pairs and we use the Thompson sampling to choose effective content, which helps to reduce randomness and ensure repeatability. We compare the attack performance of our framework with other methods, and experimental results demonstrate that our algorithm is more effective. The evasion rates of PSP-Mal against the LightGBM models trained on EMBER and SOREL-20M reach 76.88% and 72.03%, respectively.