Log lines consist of static parts that characterize their structure and enable assignment of event types, and event parameters, i.e., variable parts that provide specific information on system processes, such as host and user names, IP addresses, and file operations. Many detection approaches only focus on anomalous event type occurrences, i.e., they parse log lines to derive unique event identifiers and subsequently detect anomalies in event sequences or event count vectors, but neglect variable parts of log lines entirely during analysis. This is especially problematic, when monitoring strongly structured log data that contains only a small number of distinct event types, for example, logs that consist of strict key value pairs, i.e., parameters that occur consistently throughout all log lines, such as it is case in access and audit logs. Thus, novel approaches are required, which focus on analysis of log lines' variable parts. In this paper, we propose the variable type detector (VTD), a novel unsupervised approach that autonomously analyzes variable log line parts to enable anomaly detection. It assigns data types to each variable, which also include probability distributions for discrete and continuous variables. The VTD raises an alarm if a variable's data type changes. Furthermore, it implements a robust indicator function that reduces false positives by tracking the data type history of each variable and reports only significant data type changes. Additionally, an event indicator enables event-based anomaly detection by taking into account the data types of all variables of a single event type. The evaluation conducted on open-source log data, demonstrates the effectiveness of the VTD compared to conventional anomaly detection approaches, such as time series analysis and PCA. Consequently, the VTD acts as a solution that extends the intrusion detection capabilities of security information and event management (SIEM) and integrates with modern concepts of endpoint detection and response (EDR) and extended detection and responses (XDR), while simultaneously serving as an asset for process monitoring that supports user and entity behavior analytics (UEBA).

Behavior-based anomaly detection (AD) approaches for enterprise-IT security are not easily applicable to other domains, such as embedded devices and IoT nodes in cyber-physical systems. AD approaches are usually highly optimized for specific purposes, tightly bound to domain-specific technologies and rely on a specific syntax of investigated data. Data from cyber-physical systems is however highly diverse, often poorly documented and not easily ingested for automated analysis. AECID provides an anomaly detection approach, that monitors unstructured textual event data (i.e., log data), and implements self-learning for autonomous operation. A parser generator establishes a model of normal system behavior on top of observed events, which then can be leveraged to detect anomalies as deviations from that baseline. The unsupervised anomaly detection approaches of AECID apply machine learning techniques to perform sequence analysis, correlation analysis and statistical tests of events represented in log data. This paper discusses AECID's applicability in a building security system use case. A proof of concept demonstrates the effective detection of anomalies in log data of a building access control system stemming from card misuse, including stolen access cards and cloned cards.

Cyber criminals utilize compromised user accounts to gain access into otherwise protected systems without the need for technical exploits. User and Entity Behavior Analytics (UEBA) leverages anomaly detection techniques to recognize such intrusions by comparing user behavior patterns against profiles derived from historical log data. Unfortunately, hardly any real log data sets suitable for UEBA are publicly available, which prevents objective comparison and reproducibility of approaches. Synthetic data sets are only able to alleviate this problem to some extent, because simulations are unable to adequately induce the dynamic and unstable nature of real user behavior in generated log data. We therefore present a real system log data set from a cloud computing platform involving more than 5000 users and spanning over more than five years. To evaluate our data set for the scenario of account hijacking, we outline a method for attack injection and subsequently disclose the resulting manifestations with an adaptive anomaly detection mechanism.

Log data analysis is an essential task when it comes to understanding a computer's or a network's system behavior, and enables security analysis, fault diagnosis, performance analysis, or intrusion detection. An established technique for log analysis is log line clustering, which allows to group similar events and to detect outliers, malicious clusters or changes in system behavior. However, log line clusters usually lack meaningful descriptions that are required to understand the information provided by log lines within a cluster. Template generators allow to produce such descriptions in form of patterns that match all log lines within a cluster and therefore describe the common features of the lines. Current approaches only allow generation of token-based (e.g., space-separated words) templates, which are often inaccurate, because they do not recognize words that can be spelled differently as similar and require further information on the structure and syntax of the data, such as predefined delimiters. Consequently, novel character-based template generators are required that provide robust templates for any type of computer log data, which can be applied in security information and event management (SIEM) solutions, for continuous auditing, quality inspection and control. In this paper, we propose a novel approach for computing character-based templates, which combines comparison-based methods and heuristics. To achieve this goal, we solve the problem of efficiently calculating a multi-line alignment for a group of log lines and compute an accurate approximation of the optimal character-based template, while reducing the runtime from $O(n^m)$ to $O(mn^2)$. We demonstrate the accuracy of our approach in a detailed evaluation, applying a newly introduced measure for accuracy, the Sim-Score, which can be computed independently from a ground truth, and the established F-Score. Furthermore, we assess the robustness of the algorithm and the influence of different log data properties on the quality of the resulting templates.