Cyber threats are evolving rapidly, making anomaly detection (AD) in system log data increasingly important for detection of known and unknown attacks. The configuration of AD algorithms heavily depends on the data at hand. It often involves a complex feature selection process and the determination of parameters such as thresholds or window sizes. In many cases, configuration requires manual intervention by domain experts, which limits accessibility and effectiveness of AD algorithms. This work introduces a Configuration-Engine (CE), which employs a semi-supervised approach to automate the configuration process or optimize existing configurations. The CE utilizes statistical methods to identify log line properties to recognize meaningful tokens for AD methods to monitor. It categorizes variables by their characteristics and behavior over time, then specifies which log parts a detector should observe, and sets appropriate configuration parameters.The CE was evaluated using four different detectors. Evaluations on different Apache Access and audit datasets containing attack traces showed that the CE achieved an average precision of over 0.94 for Apache and over 0.79 for audit datasets, while maintaining high recall, competing with the performance of expert-crafted configurations. The optimization approach was able to strongly improve the precision of both the CE’s and the experts’ configurations for Apache data in 7 out of 16 cases. Furthermore, the CE’s configurations were significantly dissimilar to each other when generated on audit data, highlighting the importance of automated configuration.

The rapid advancement of Large Language Models (LLMs) has transformed natural language processing, yet comprehensive evaluation methods are necessary to ensure their reliability, particularly in Retrieval-Augmented Generation (RAG) tasks. This study aims to evaluate and compare the performance of open-source LLMs by introducing a rigorous evaluation framework. We benchmark 20 LLMs using a combination of established metrics such as BLEU, ROUGE, BERTScore, along with and a novel metric, RAGAS. The models were tested across two distinct datasets to assess their text generation quality. Our findings reveal that models like nous-hermes-2-solar-10.7b and mistral-7b-instruct-v0.1 consistently excel in tasks requiring strict instruction adherence and effective use of large contexts, while other models show areas for improvement. This research contributes to the field by offering a comprehensive evaluation framework that aids in selecting the most suitable LLMs for complex RAG applications, with implications for future developments in natural language processing and big data analysis.

Cyber criminals utilize compromised user accounts to gain access into otherwise protected systems without the need for technical exploits. User and Entity Behavior Analytics (UEBA) leverages anomaly detection techniques to recognize such intrusions by comparing user behavior patterns against profiles derived from historical log data. Unfortunately, hardly any real log data sets suitable for UEBA are publicly available, which prevents objective comparison and reproducibility of approaches. Synthetic data sets are only able to alleviate this problem to some extent, because simulations are unable to adequately induce the dynamic and unstable nature of real user behavior in generated log data. We therefore present a real system log data set from a cloud computing platform involving more than 5000 users and spanning over more than five years. To evaluate our data set for the scenario of account hijacking, we outline a method for attack injection and subsequently disclose the resulting manifestations with an adaptive anomaly detection mechanism.

Intrusion Detection Systems (IDS) rely on the availability and correctness of Indicators of Compromise (IoC), i.e., artifacts such as IP addresses that are known to correspond to malicious system activities. However, the simple nature and limited validity of these indicators impairs protection against cyber threats. Tactics, Techniques and Procedures (TTP) provide abstract information on attacker behavior, but are only available in human-readable format that prevents automatic detection using IDSs. In this paper we therefore propose an approach that extracts cyber threat intelligence from raw log data and combines the advantages of IoCs and TTPs by producing detectable patterns of complex system behavior. Other than existing approaches, our approach employs log data anomaly detection to disclose suspicious log events, which are used for iterative clustering, pattern recognition, and refinement. Our evaluations show that automatically extracted threat intelligence corresponding to a multi-step attack is suitable for detection of the same attack on another system.