Reverse Engineering (RE) is a critical task performed by security professionals for various purposes. However, the complexity and exertion of malware reverse engineering, particularly for ransomware, have posed significant challenges to experts in the field. In response, this study explores the feasibility of incorporating deep learning techniques to assist the ransomware reverse engineering (RE). To tackle specific challenges of encryption loop recognition, our approach employs two learning strategies. Firstly, we develop code-obfuscation-resilient and encryption-algorithm-agnostic features, including K-complexity and operations that yield equiprobable outputs. Secondly, we carefully select a neural network architecture capable of extracting informative features. The evaluation of our toolchain shows that our toolchain achieves an accuracy of 99% on the test set. Our method exhibits strong generalization capabilities, as it successfully handled common code obfuscation schemes, proprietary and unknown ciphers. When applied to real-world ransomware samples such as WannaCry, Conti, Lockbit, and TeslaCryt, our toolchain effectively identified 205 encryption loops with a low false positive rate of 6.8%. These findings validate the effectiveness of our approach in automatically recognizing encryption code during ransomware reverse engineering.

Abstract Binary analysis, the process of examining software without its source code, plays a crucial role in understanding program behavior, e.g., evaluating the security properties of commercial software, and analyzing malware. One challenging aspect of this process is to classify data encoding schemes, such as encryption and compression, due to the absence of high-level semantic information. Existing approaches either rely on code similarity, which only works for known schemes, or heuristic rules, which lack scalability. In this paper, we propose DESCG, a novel deep learning-based method for automatically classifying four widely employed kinds of data encoding schemes in binary programs: encryption, compression, decompression, and hashing. Our approach leverages dynamic analysis to extract execution traces from binary programs, builds data dependency graphs from these traces, and incorporates critical feature engineering. By combining the specialized graph representation with the Graph Neural Network (GNN), our approach enables accurate classification without requiring prior knowledge of specific encoding schemes. The Evaluation result shows that DESCG achieves 97.7% accuracy and an F1 score of 97.67%, outperforming baseline models. We also conducted an extensive evaluation of DESCG to explore which feature is more important for it and examine its performance and overhead.

Abstract Deep learning techniques have achieved remarkable performance in wide-ranging tasks. However, when trained on privacy-sensitive datasets, the model parameters may expose private information in training data. Prior attempts for differentially private training, although offering rigorous privacy guarantees, lead to much lower model performance than the non-private ones. Besides, different runs of the same training algorithm produce models with large performance variance. To address these issues, we propose DPlis– Differentially Private Learning wIth Smoothing. The core idea of DPlis is to construct a smooth loss function that favors noise-resilient models lying in large flat regions of the loss landscape. We provide theoretical justification for the utility improvements of DPlis. Extensive experiments also demonstrate that DPlis can effectively boost model quality and training stability under a given privacy budget.