Clarifying malware communication protocols is critical for enhancing system security. Existing protocol reverse engineering (PRE) methods lack effective strategies, failing to recover protocol structures or infer precise field semantics. To address these challenges, we propose MALPRE, an execution trace-based PRE framework that integrates precise program analysis with large language models (LLMs) for automated malware protocol format recovery. MALPRE first embeds and hierarchically clusters the code slices to restore message formats. It then introduces a multi-agentic workflow comprising code analyst, malware expert, and protocol puzzler roles to collaboratively infer field semantics. Evaluations on the dataset containing six popular malware frameworks demonstrate that MALPRE outperforms the state-of-the-art methods—including four conventional tools (e.g., BINPRE) and three LLM-based approaches (e.g., DEGPT)—by 23.8% (F1) in field structure recovery and 4.8% (FSS-score) in semantic inference. MALPRE has successfully analyzed APT backdoor communications and emerging botnets, with two extracted traffic rules assigned by Open ET Ruleset.

Variable name recovery is a fundamental task in software reverse engineering, crucial for various cybersecurity applications including binary code understanding, protocol format recovery, and malware analysis. The core challenge lies in the semantic loss during compilation to binary code. Although intelligence-driven methods have shown progress, several issues persist: generated names remain vague, semantic inconsistency occurs across different contexts, and inference ordering lacks systematic planning. In this paper, we propose VarAgent, a variable naming agent driven by two domain-enhanced large language models: one focuses on inferring variable names within functions through reinforcement learning from reverse-engineering feedback, while the other propagates and fuses semantics across variables using graph neural networks embedded within the LLM. We also design a confusion-guided planner that mimics expert reverse engineers’ analysis strategies. Experimental results demonstrate that VarAgent achieves average precision and F1 scores of 34.9% and 34.7% on classic and novel datasets, outperforming state-of-the-art tools by 4.0%–15.4% (including ReSym, GenNm, DeGPT, and VarBERT). We further demonstrate its effectiveness in supporting downstream tasks, including protocol field inference, binary semantic search, and malware summarization.

Software reverse analysis is a key technology in the field of cyber-security. With the increasing scale and complexity of software, this technology is facing great challenges. Binary code modularization (BCM), as the basic work of software reverse, plays an important role in extracting semantics, narrowing the analysis scope and locating key position. The semantics of strings underlying code is a significant hint, with being processed using natural language processing and artificial intelligence technology help to reverse analysis effectively. However, most of the existing modularization methods ignore these semantics, which limits the in-depth understanding of binary code. This paper proposes a semantic-driven reverse engineering framework for binary code modularization (SBCM). Firstly, the rich string is extracted from the binary file into a large language model for semantics analysis. Then, the semantic information of the string is combined with the control flow graph to construct the function semantic graph (FSG). Subsequently, a function summary is generated based on the FSG. Finally, semantic embedding is generated for Summaries and semantic-driven integrated clustering is carried out to realize binary code modularization. The experiment results show that SBCM improves the F1 value by 12.6% on average compared with the existing methods, which proves its effectiveness and superiority in binary code modularization.