Recent advances in LLM agentic systems have improved the automation of offensive security tasks, particularly for Capture the Flag (CTF) challenges. We systematically investigate the key factors that drive agent success and provide a detailed recipe for building effective LLM-based offensive security agents. First, we present CTFJudge, a framework leveraging LLM as a judge to analyze agent trajectories and provide granular evaluation across CTF solving steps. Second, we propose a novel metric, CTF Competency Index (CCI) for partial correctness, revealing how closely agent solutions align with human-crafted gold standards. Third, we examine how LLM hyperparameters, namely temperature, top-p, and maximum token length, influence agent performance and automated cybersecurity task planning. For rapid evaluation, we present CTFTiny, a curated benchmark of 50 representative CTF challenges across binary exploitation, web, reverse engineering, forensics, and cryptography. Our findings identify optimal multi-agent coordination settings and lay the groundwork for future LLM agent research in cybersecurity.

The proliferation of advanced detection techniques and the evolution of next-generation firewalls and antivirus engines have led to the increasing sophistication of cyber threats. In this context, malware authors are crafting disguised payloads that mimic benign behavior. To achieve this, the use of Windows signed executables/binaries (Living off the Land Binaries or LOLBins) and libraries has become increasingly relevant as a method to evade antivirus and signature-based detection techniques. These binaries inherently grant attackers a level of trust within the Windows operating system as they are signed by Microsoft. Understanding the specific LOLBins used by different attacks and their variants is crucial for developing effective detection rules and enhanced threat intelligence. Therefore, in this work, we analyze the presence of LOLBins from five distinct cyber attacks through dynamic analysis to determine the ubiquity and role of these Windows signed binaries in these attacks. We observe that the usage of LOLBins is nearly 51% of the payloads across Ransomware, Cryptominers, Advanced Persistent Threats (APTs), Information Stealers, and Remote Access Trojans (RATs)/Trojans. We also identify the distinct roles of the same LOLBins in attack variants in terms of evading defense strategies, downloading payloads, and offering stealth. Notably, ransomware and crypto miner payloads exhibit a higher diversity of utilizing 55 and 30 distinct LOLBins, respectively. Finally, we systematically analyze and compare the usage of LOLBins in Cobalt Strike payloads—a legacy multipurpose tool used by many malware families to evade detection, gather information, and persist within the victim environment. We identify Cobalt Strike to have the highest usage among all categories, at almost 73%.