Detecting lateral movement threats in large-scale system logs is a critical challenge due to the scarcity of labelled attack data, the presence of imbalanced datasets, and the sophisticated nature of modern adversaries. To address these issues, we propose ConceptUML , a semantic-driven, fully unsupervised threat detection framework designed to automatically identify anomalies related to lateral movement in heterogeneous log data. ConceptUML is structured around a three-phase architecture. In Phase 1 (Latent Semantic Learning) , contextualized embeddings generated by Sentence-BERT are combined with Non-negative Matrix Factorization to extract abstract concepts from system logs and external threat intelligence sources such as MITRE ATT&CK and CAPEC. In Phase 2 (Unsupervised Threat Detection) , a Hidden Markov Model is applied to cluster logs based on learned concepts, and each cluster is scored according to its semantic similarity to known adversarial techniques. Phase 3 (Decision Refinement) uses topic modelling to further isolate malicious event log subsets from within suspicious clusters, enabling high-precision triage. We evaluate ConceptUML using four real-world event log datasets, including Windows Event Logs and multiple subsets of the LMD-23 dataset, encompassing attacks such as exploitation of hashing techniques and remote services. The enhanced model with topic modelling achieves up to 92.54% detection quality and reduces detection error to as low as 8.14%, outperforming several baseline approaches including AutoEncoder, LogAnomaly, LOF, and DBScan. Our results confirm that ConceptUML delivers interpretable, scalable, and highly effective detection of lateral movement threats without requiring labelled training data or extensive manual feature engineering.

The escalating prevalence of Advanced Persistent Threats (APTs) necessitates the development of more robust solutions capable of effectively thwarting these attacks by monitoring system activities across individual hosts. Existing cloud-native security applications utilize a combination of rule-based and machine learning-based detection techniques to protect digital assets . However, these approaches have limitations. Rule-based detection depends on predefined rules to identify specific attack patterns. Persistent attackers can often evade detection by carefully ensuring that their behavior circumvents these rules. In contrast, machine learning-based detection techniques, which learn attack patterns from data, rely heavily on the availability of labeled data for training. However, labeled data is often unavailable and can be labor-intensive and costly to obtain. In this paper, we address the challenge of detecting APT attacks more holistically by leveraging attackers’ behavior during communication with Command and Control (C2) servers, a critical phase observed in most APT attacks. We aim to reduce false positive alerts for threat hunters by analyzing system network logs to detect potential network beaconing, a common attribute of various malware . We introduce a novel hybrid approach, called NetSpectra Sentinel , which employs a Continuous Time Hidden Markov Model (CT-HMM) to detect hidden states underlying observed patterns within the network logs and Time Series Decomposition (TSD) to model temporal patterns. We evaluate the effectiveness of our approach using 14 benchmark datasets and one synthetic dataset , comparing our method with other state-of-the-art statistical-based and botnet detection techniques. The results demonstrate that our technique achieves significantly higher accuracy in most cases, and even when existing techniques fail, our approach can still detect beaconing post-initial compromise with up to 90% accuracy. Additionally, we achieve up to four times better performance in terms of precision compared to existing statistical-based techniques.

In the rapidly changing cybersecurity landscape, threat hunting has become a critical proactive defense against sophisticated cyber threats. While traditional security measures are essential, their reactive nature often falls short in countering malicious actors’ increasingly advanced tactics. This paper explores the crucial role of threat hunting, a systematic, analyst-driven process aimed at uncovering hidden threats lurking within an organization's digital infrastructure before they escalate into major incidents. Despite its importance, the cybersecurity community grapples with several challenges, including the lack of standardized methodologies, the need for specialized expertise, and the integration of cutting-edge technologies like artificial intelligence (AI) for predictive threat identification. To tackle these challenges, this survey paper offers a comprehensive overview of current threat hunting practices, emphasizing the integration of AI-driven models for proactive threat prediction. Our research explores critical questions regarding the effectiveness of various threat hunting processes and the incorporation of advanced techniques such as augmented methodologies and machine learning. Our approach involves a systematic review of existing practices, including frameworks from industry leaders like IBM and CrowdStrike. We also explore resources for intelligence ontologies and automation tools. The background section clarifies the distinction between threat hunting and anomaly detection, emphasizing systematic processes crucial for effective threat hunting. We formulate hypotheses based on hidden states and observations, examine the interplay between anomaly detection and threat hunting, and introduce iterative detection methodologies and playbooks for enhanced threat detection. Our review encompasses supervised and unsupervised machine learning approaches, reasoning techniques, graph-based and rule-based methods, as well as other innovative strategies. We identify key challenges in the field, including the scarcity of labeled data, imbalanced datasets, the need for integrating multiple data sources, the rapid evolution of adversarial techniques, and the limited availability of human expertise and data intelligence. The discussion highlights the transformative impact of artificial intelligence on both threat hunting and cybercrime, reinforcing the importance of robust hypothesis development. This paper contributes a detailed analysis of the current state and future directions of threat hunting, offering actionable insights for researchers and practitioners to enhance threat detection and mitigation strategies in the ever-evolving cybersecurity landscape.

We propose a novel end-to-end trust management framework for crowdsourced Internet of Things (IoT) services. The framework targets three main aspects:trust assessment,trust information credibility and accuracy, andtrust information storage. We harness theusage patternsof IoT consumers to offer a trust assessment thatadaptsto IoT consumers’ uses. Additionally, our framework ascertains thecredibilityandaccuracyof trust-related information before trust assessment. This is achieved by validating the data collected by IoT consumers and providers. In addition, our framework ensures thecontextual fairnessbetween IoT services and trust information. Moreover, we propose a blockchain-based trust information storage approach. Our proposed storage solution preserves theintegrityandavailabilityof trust information.

We introduce the concept ofadaptive trustin crowdsourced IoT services. It is a customized fine-grained trust tailored for specific IoT consumers.Usage patternsof IoT consumers are exploited to provide an accurate trust value for service providers. A noveladaptive trust management frameworkis proposed to assess the dynamic trust of IoT services. The framework leverages a novel detection algorithm to obtaintrust indicatorsthat are likely to influence the trust level of a specific IoT service type. Detected trust indicators are then used to buildservice-to-indicatormodel to evaluate a service’strust at each indicator. Similarly, ausage-to-indicatormodel is built to obtain theimportance of each trust indicatorfor a particular usage scenario. The per-indicator trust and the importance of each trust indicator are utilized to obtain an overall value of a given service for a specific consumer. We conduct a set of experiments on a real dataset to show the effectiveness of the proposed framework.

We propose a novel generic trust management framework for crowdsourced IoT services. The framework exploits amulti-perspective trust modelthat captures the inherent characteristics of crowdsourced IoT services. Each perspective is defined by a set ofattributesthat contribute to the perspective's influence on trust. The attributes are fed into a machine-learning-based algorithm to generate atrust modelfor crowdsourced services in IoT environments. We demonstrate the effectiveness of our approach by conducting experiments on real-world datasets.

We develop a novel framework for efficiently and effectively discovering crowdsourced services thatmovein close proximity to a user over a period of time. We introduce a moving crowdsourced service model which is modelled as a moving region. We propose a deep reinforcement learning-based composition approach to select and compose moving IoT services considering quality parameters. Additionally, we develop a parallel flock-based service discovery algorithm as a ground-truth to measure the accuracy of the proposed approach. The experiments on two real-world datasets verify the effectiveness and efficiency of the deep reinforcement learning-based approach.

We propose a novel distributed integrity-preserving framework for storing trust information in crowdsourced IoT environments. The integrity and availability of the trust information is paramount to ensure accurate trust assessment. Our proposed framework leverages the blockchain to build a distributed storage medium for trust-related information that ensures its integrity. We propose a geo-scoping approach, which ensures that trust-related information is only available where needed, thus, enabling fast access and storage space preservation. We conduct several experiments using real datasets to highlight the effectiveness of our framework.

We propose just-in-time memoryless trust for crowdsourced IoT services. We leverage the characteristics of the IoT service environment to evaluate their trustworthiness. A novel framework is devised to assess a service's trust without relying on previous knowledge, i.e., memoryless trust. The framework exploits service-session-related data to offer a trust value valid only during the current session, i.e., just-in-time trust. Several experiments are conducted to assess the efficiency of the proposed framework.

We introduce the notion of Adaptive Trust in crowdsourced IoT services; a usage-based trust that represents a service's trustworthiness based on consumers' uses. A novel four-stage framework is proposed to assess the dynamic service trust by leveraging how the service is being used. The first stage uses an algorithm to predict different trust factors that affect the overall trustworthiness of an IoT service. Trust factors are fed to the second stage to build a service-to-factor model that predicts the trustworthiness of a service at each given trust factor. A usage-to-factor model is built at the third stage, which detects the importance of each factor for a specific usage scenario. The last stage utilizes the two models to compute a trust value specifically tailored for a particular usage. Several experiments have been conducted using real dataset to ensure the efficiency of the proposed approach.