Abstract The widespread adoption of encryption in network traffic significantly challenges traditional detection methods that rely on payload analysis. Existing approaches often convert traffic into images or sequences for deep learning models, producing redundant features and struggling with multi-protocol environments. In this study, we propose HEAT (Header-Embedded Attention for Traffic Detection), a novel model that leverages packet header fields to develop a robust characteristic representation for encrypted traffic analysis. HEAT introduces a hierarchical attention mechanism combined with a novel contextual embedding technique that enhances the semantic representation of header field values. Additionally, HEAT integrates an adapted Kolmogorov–Arnold Network classifier with B-spline activations and L1 weight regularization, optimizing the model for efficient real-time processing. Extensive evaluations on CICIDS-2018, Stratosphere, and ISCX2012 datasets demonstrate HEAT’s superior performance, achieving 98.95% accuracy and 98.28% F1-score on CICIDS-2018, 99.5% accuracy and 98.54% F1-score on Stratosphere, and 99.75% accuracy with 99.25% F1-score on ISCX2012. HEAT significantly outperforms CNN, LSTM, and BiGRU baselines. Moreover, it maintains detection accuracy above 98.95% during incremental learning, with only a 0.9% F1-score drop, compared with 6.55% in conventional models. These results highlight HEAT’s novelty, stability, and adaptability, making it a scalable and robust solution for encrypted malicious traffic detection.

The growing prevalence of encrypted malicious network traffic poses significant challenges for cybersecurity, as it conceals the content from traditional detection methods. Temporal convolutional networks (TCNs) present promising capabilities for extracting complex temporal features and patterns from the dynamic traffic flow data. However, the unidirectional nature of traditional TCNs limits their effectiveness in capturing the full context of network traffic, which often exhibits bidirectional temporal dependencies. Consequently, a few studies have proposed bidirectional TCN (BiTCN) architectures to address the limitations. However, these methods present complex architectures that require a significant amount of parameters to be learned, which imposes high memory requirements on the computational resources for training such models. In this study, we introduce the efficient bidirectional TCN (eBiTCN) model, an efficient BiTCN that requires fewer parameters yet not at the expense of computational cost and effective detection. The eBiTCN framework combines a bidirectional processor, a lightweight gating mechanism, temporal attention, dropout, a novel loss function, and dense layers. Extensive experiments show that eBiTCN outperforms eight state-of-the-art competing models in terms of detection efficacy, speed, and scalability. The eBiTCN model showcased robust performance in detecting evolving attacks and excelled across various real-world datasets. Its efficiency in training speed and reduced memory usage translates to lower infrastructure costs, making it an accessible and effective choice for deployment. These findings highlight eBiTCN’s practicality and dependability in addressing contemporary network security needs.

Detecting malicious network traffic in large-scale, dynamic environments presents a significant challenge due to the complexity of network relationships and the evolving nature of cyber threats. Existing graph-based and sequence-based models often fail to capture both spatial dependencies and temporal patterns effectively, resulting in suboptimal detection. This study introduces the Multi-view Graph Adaptive Network (MGAN), a novel framework that integrates multi-hop graph neural network (GNN) aggregation with transformer-based sequence modeling to address these challenges. MGAN captures long-range spatial dependencies and temporal dynamics in network traffic, enabling the detection of complex attack patterns. It incorporates Dirichlet sampling for robust neighbor selection in sparse and noisy data environments and mutual information maximization to align multi-view representations for consistency. Additionally, a multi-view attention mechanism aggregates information across different hops, balancing local and global network context. Extensive experiments on four real-world datasets demonstrate MGAN’s superiority over 7 baseline models, achieving an average F1-Score above 97%, surpassing the best baseline by 2.35%. MGAN maintains detection accuracy above 97% and remains robust under data sparsity, achieving F1-Scores over 95% even when 40% of connectivity information is removed. Under noisy conditions, MGAN retains accuracy above 93%, outperforming baselines by over 4.5%. In zero-day attack scenarios, it achieves detection rates exceeding 96% for previously unseen attack categories. MGAN also exhibits exceptional computational efficiency, processing 2,034 samples per second with a detection time of 3.00 milliseconds per sample, outperforming all competing models in both accuracy and speed.

With the rapid advancements in internet technology, the complexity and sophistication of network traffic attacks are increasing, making it challenging for traditional anomaly detection systems to analyze and detect malicious network attacks. The increasing advancedness of cyber threats calls for innovative approaches to identify malicious patterns within network traffic precisely. The primary issue lies in the fact that these approaches do not focus on the essential adaptive features of network traffic. We proposed an effective anomaly detection system for malicious network traffic attacks called the Deep Ensemble Learning Model (DELM). We leverage the structure of the Feedforward Deep Neural Network (FDNN), and Deep Belief Network (DBN), incorporating multiple hidden layers with non-linear activation functions. Integrating Adaptive Feature Aggregation (AFA) with the FDNN algorithm dynamically adjusts the feature aggregation process based on incoming traffic characteristics to improve adaptability. The Conditional Generative Network was employed to enhance DELM for generating data for minority classes. To improve the model’s accuracy, we applied batch normalization and data augmentation techniques for preprocessing, utilized n-gram, one-hot encoding, and feature aggregation methods for effective feature extraction. This study significantly contributes to network security by enhancing systems for detecting malicious network traffic. With its interpretability and adaptability, our proposed model shows promise in addressing the evolving cyber threat and fortifying critical network infrastructure. The experimental results demonstrate that our model performs with higher stability than the existing state-of-the-art detection approaches, as reflected by its higher accuracy, precision, recall, F1-score, and AUC-ROC.