The emergence of DNS-over-HTTPS (DoH) tunnels poses significant challenges to network security, particularly when encountering unknown traffic patterns not seen during training. Existing approaches struggle to effectively identify novel DoH tunnel variants while maintaining accurate classi-fication of known traffic patterns. In this paper, we propose DualConBound, a novel framework that combines dual-view contrastive learning with dynamic feature boundary estimation to address open-set network traffic classification. Our approach leverages complementary traffic representations and adaptive decision boundaries to better distinguish between known and unknown traffic patterns. Through extensive experiments on real-world network traffic datasets, we demonstrate that our framework achieves 91 % accuracy on known traffic patterns and 71 % accuracy on unknown variants, outperforming other methods by 6% in open-set scenarios. Our work provides a robust solution for identifying emerging DoH tunnel threats in real-world network environments and establishes a new paradigm for open-set network traffic classification.

In the field of cybersecurity, encrypted traffic classification is of paramount importance, serving as a crucial technology for ensuring data privacy and defending against cyber attacks. Existing ETC (Encrypted Traffic Classification) methods commonly face three challenges in real-world Mobile Internet environments, particularly pertinent to the multimedia context: 1) singular entity traffic information, 2) fixed-size feature filters in learning models, and 3) reliance on traditional encryption protocols. To address these, we introduce the innovative Packet-Flow Dual Views (PFDV) framework. PFDV aims to push the performance boundaries of ETC tasks in multimedia traffic scenarios by examining encrypted payloads separately from the packet and flow views. More specifically, PFDV employs a dual attention framework, comprising both payload attention and gram attention, to accurately identify nuanced differences in network traffic. Such an approach underscores granular packet-level details while offering a comprehensive flow view with packet positional data, striving for an optimal balance in multimedia traffic analysis. PFDV demonstrated superior performance across diverse tasks, achieving accuracy rates and F1 scores significantly higher than competing methods, with results of up to 98.34% accuracy and 95.69 % F1 score in multimedia traffic classification. Furthermore, ablation studies demonstrate each component in PFDV contributes significantly to the overall enhancement of ETC performance in the model.

The widespread adoption of DNS over HTTPS (DoH) has introduced significant challenges in network security, particularly the emergence of DoH tunnels. Existing methods, reliant on large labeled datasets, struggle to adapt to novel DoH tunnel variants. We present ProtoDoH, a novel framework for DoH tunnel detection that uniquely integrates prototypical networks with meta-learning. Our method leverages few-shot learning principles to detect DoH tunnels with minimal training samples, addressing the challenge of sample scarcity in new attack scenarios. By employing a metric-based meta-learning framework, ProtoDoH enables rapid adaptation to novel DoH tunnel variants, significantly reducing the detection time for new DoH tunnels. Experimental results demonstrate that our approach achieves over 99% accuracy in detecting DoH tunnels with as few as 5 samples, notably outperforming traditional machine learning and deep learning methods. Furthermore, our model exhibits strong generalization capabilities across different network environments and DoH tunnel tools.

Traditional Encrypted Traffic Classification (ETC) methods face a significant challenge in classifying large volumes of encrypted traffic in the open-world assumption, i.e., simultaneously classifying the known applications and detecting unknown applications. We propose a novel Open-World Contrastive Pre-training (OWCP) framework for this. OWCP performs contrastive pre-training to obtain a robust feature representation. Based on this, we determine the spherical mapping space to find the marginal flows for each known class, which are used to train GANs to synthesize new flows similar to the known parts but do not belong to any class. These synthetic flows are assigned to Softmax's unknown node to modify the classifier, effectively enhancing sensitivity towards known flows and significantly suppressing unknown ones. Extensive experiments on three datasets show that OWCP significantly outperforms existing ETC and generic open-world classification methods. Furthermore, we conduct comprehensive ablation studies and sensitivity analyses to validate each integral component of OWCP.