Large Language Models (LLMs) have been explored for automating or enhancing penetration testing tasks, but their effectiveness and reliability across diverse attack phases remain open questions.This study presents a comprehensive evaluation of multiple LLM-based agents, ranging from singular to modular designs, across realistic penetration testing scenarios, analyzing their empirical performance and recurring failure patterns.We further investigate the impact of core functional capabilities on agent success, operationalized through five targeted augmentations: Global Context Memory (GCM), Inter-Agent Messaging (IAM), Context-Conditioned Invocation (CCI), Adaptive Planning (AP), and Real-Time Monitoring (RTM).These interventions respectively support the capabilities of Context Coherence & Retention, Inter-Component Coordination & State Management, Tool Usage Accuracy & Selective Execution, Multi-Step Strategic Planning & Error Detection & Recovery, and Real-Time Dynamic Responsiveness.Our findings reveal that while some architectures natively exhibit select properties, targeted augmentations significantly enhance modular agent performance-particularly in complex, multi-step, and real-time penetration testing scenarios.

The proliferation of open-sourced Large Language Models (LLMs) and diverse downstream tasks necessitates efficient model selection, given the impracticality of fine-tuning all candidates due to computational constraints. Despite the recent advances in LLM selection, a fundamental research question largely remains nascent: how can we model the dynamic behaviors of LLMs during fine-tuning, thereby enhancing our understanding of their generalization performance across diverse downstream tasks? In this work, we propose a novel theoretical framework that provides a proper lens to assess the generalization capabilities of LLMs, thereby enabling accurate and efficient LLM selection for downstream applications. In particular, we first derive a PAC-Bayesian Generalization Bound that unveils fine-tuning dynamics of LLMs and then introduce LensLLM, a Neural Tangent Kernel (NTK)-based Rectified Scaling Model that enables accurate performance predictions across diverse tasks while maintaining computational efficiency. Extensive empirical results on 3 large-scale benchmarks demonstrate that our model achieves up to 91.1% accuracy and reduces up to 88.5% computational cost in LLM selection, outperforming 5 state-of-the-art methods. We open-source our proposed LensLLM model and corresponding results at LensLLM.io.

Non-IID transfer learning on graphs is crucial in many high-stakes domains. The majority of existing works assume stationary distribution for both source and target domains. However, real-world graphs are intrinsically dynamic, presenting challenges in terms of domain evolution and dynamic discrepancy between source and target domains. To bridge the gap, we shift the problem to the dynamic setting and pose the question: given the *label-rich* source graphs and the *label-scarce* target graphs both observed in previous $T$ timestamps, how can we effectively characterize the evolving domain discrepancy and optimize the generalization performance of the target domain at the incoming $T+1$ timestamp? To answer it, we propose a generalization bound for *dynamic non-IID transfer learning on graphs*, which implies the generalization performance is dominated by domain evolution and domain discrepancy between source and target graphs. Inspired by the theoretical results, we introduce a novel generic framework named EvoluNet. It leverages a transformer-based temporal encoding module to model temporal information of the evolving domains and then uses a dynamic domain unification module to efficiently learn domain-invariant representations across the source and target domains. Finally, EvoluNet outperforms the state-of-the-art models by up to 12.1%, demonstrating its effectiveness in transferring knowledge from dynamic source graphs to dynamic target graphs.

While machine learning models have demonstrated the ability to detect cyber attacks, their deployment to operational scenarios is often limited due to the possibility of the model failing to detect attacks not included in the training set. Transfer learning has been shown to be a possible response to this common problem in the cyber domain. However, there are many operational questions that must be addressed before transfer learning can be implemented. First, and foremost, is the question of how to detect a change in an operational environment (e.g., the presence of a new attack variant). Transfer distance is a measure of dissimilarity between two learning problems. This study evaluates common distance metrics that could be used to estimate transfer distance between cyber attack learning problems, proposes a new method for calculating a multivariate transfer distance measure, and formulates a method for monitoring streaming data for changes in the environment using a one class Naïve Bayes model. These techniques are tested and evaluated on a publicly available cyber attack detection data set that contains multiple attack types.

Reinforcement learning (RL) operating on attack graphs leveraging cyber terrain principles are used to develop reward and state associated with determination of surveillance detection routes (SDR). This work extends previous efforts on developing RL methods for path analysis within enterprise networks. This work focuses on building SDR where the routes focus on exploring the network services while trying to evade risk. RL is utilized to support the development of these routes by building a reward mechanism that would help in realization of these paths. The RL algorithm is modified to have a novel warm-up phase which decides in the initial exploration which areas of the network are safe to explore based on the rewards and penalty scale factor.

Active learning is a practical field of machine learning that automates the process of selecting which data to label. Current methods are effective in reducing the burden of data labeling but are heavily model-reliant. This has led to the inability of sampled data to be transferred to new models as well as issues with sampling bias. Both issues are of crucial concern in machine learning deployment. We propose active learning methods utilizing combinatorial coverage to overcome these issues. The proposed methods are data-centric, as opposed to model-centric, and through our experiments we show that the inclusion of coverage in active learning leads to sampling data that tends to be the best in transferring to better performing models and has a competitive sampling bias compared to benchmark methods.

Metric learning is a well-developed field in machine learning and has seen recent application in the area of prognostics and health management (PHM). Metric learning allows for fault diagnosis or condition monitoring models to be developed with the assumption that a machine- or load-specific similarity metric can be learned after model deployment. Existing literature has used metric learning to fine-tune deep learning models to address machine-to-machine differences and differences in working conditions. Here, we study metric learning in isolation, not as an intermediate step in deep learning, by conducting a comparative study of Principal Component Analysis (PCA), Neighborhood Component Analysis (NCA), Local Fisher Discriminant Analysis (LFDA), and Large Margin Nearest Neighbor (LMNN). We consider performance metrics for prediction performance, cluster performance, feature sensitivity, sample efficiency, and latent space efficiency. We find that linear partitions on the latent spaces learned via metric learning are able to achieve accuracies greater than 90% on Case Western Reserve University’s bearing fault data set using only the drive-end vibration signal. We find PCA to be dominated by metric learning algorithms for all working loads considered. And, in sum, we demonstrate classical metric learning algorithms to be a promising approach for learning machine-and load-specific similarity metrics for PHM with minor data processing and small samples.

Supervised learning algorithms require a set of labeled training data. In many engineering applications, acquiring and accurately labeling the training data can be time consuming, burdensome, and costly. Active learning is an area of machine learning that selects observations in an unlabeled set to be passed to an oracle to retrieve the ground truth label, thereby improving the efficiency of the labeling and training process. However, most active learning algorithms only consider model improvement and, therefore, ignore cost considerations. The active learning algorithms that do consider cost require the practitioner to specify the trade-off between model improvement and cost. We propose an active learning with cost method that does not require this trade-off to be specified by randomly sampling observations from the Pareto optimal frontier. Further, we propose an extension to this method that accounts for uncertainty in the cost estimate of labeling an observation. These methods are evaluated on publicly available data sets, and the numerical experiments demonstrate that the proposed methods can produce models that achieve similar performance to standard active learning algorithms while reducing the labeling cost.