Generative malicious traffic dectection model based on heterogeneous graph attention network and cross-type distillation
Online published: 2026-05-06
Copyright
With the increasing severity of cybersecurity threats, encrypted traffic dectection has become a core challenge in the field of network security. Graph neural networks, with their powerful structural modeling capabilities, have shown significant potential in capturing complex correlation patterns among encrypted traffic byte units. However, traditional homogeneous graph modeling methods struggle to comprehensively characterize the multidimensional heterogeneous relationships between headers and payloads in encrypted traffic. While heterogeneous graph neural networks can address such complexity, independent edge-type modeling leads to parameter redundancy and a sharp increase in training overhead, coupled with a lack of collaborative knowledge transfer mechanisms between edge types. To address this, this paper proposes a cross-type distillation mechanism that constructs a unified structural representation path to enable bidirectional knowledge transfer between heterogeneous edge types, breaking the information silo effect of traditional heterogeneous graph neural network and significantly enhancing the feature expression capability of weak semantic edges. Building on this, this paper designs a lightweight malicious traffic detection model, HEDGAT (Heterogeneous Edge-aware Distilled Graph Attention Model). This model employs an edge-type weight-sharing mechanism to compress the propagation weights of multiple edge types into a unified representation space, integrating graph attention computation to significantly reduce training time while preserving the ability to perceive structural differences. Based on a heterogeneous graph framework, HEDGAT can precisely capture complex dependencies between byte units and between headers and payloads, combining a dynamic fusion mechanism to generate comprehensive traffic representations. Packet-level and flow-level classification experiments on multiple encrypted traffic datasets demonstrate that HEDGAT outperforms existing methods in classification accuracy, training efficiency, and model parameter scale, exhibiting superior performance and lightweight advantages.
Yu Kun , Lu Jiazhong , Zhang Feng , Liu Xiaolei . Generative malicious traffic dectection model based on heterogeneous graph attention network and cross-type distillation[J]. Journal of Cybersecurity, 2025 , 3(6) : 68 -79 . DOI: 10.20172/j.issn.2097-3136.250605
表 1 模型在3个数据集上的性能对比Table 1 Performance comparison of the models on 3 datasets |
| 数据集 | 模型 | 准确率 | F1 | 收敛epoch |
| GenAI | AppScanner | 70.5% | 70.6% | 42 |
| BFCN | 96.8% | 96.7% | 40 | |
| ATVITSC | 91.9% | 91.5% | 39 | |
| CMTSNN | 92.5% | 91.3% | 40 | |
| METAROCKETC | 96.7% | 94.8% | 33 | |
| TFE-GNN | 95.2% | 95.3% | 37 | |
| TFG-GNN | 96.5% | 96.8% | 34 | |
| HEDGAT | 98.1% | 98.1% | 28 | |
| ICS- GenMal | AppScanner | 66.8% | 65.5% | 48 |
| BFCN | 96.5% | 97.1% | 35 | |
| ATVITSC | 94.2% | 92.8% | 31 | |
| CMTSNN | 93.1% | 91.6% | 41 | |
| METAROCKETC | 96.6% | 96.5% | 36 | |
| TFE-GNN | 95.3% | 94.8% | 39 | |
| TFG-GNN | 96.0% | 95.6% | 34 | |
| HEDGAT | 97.9% | 98.0% | 26 | |
| WGAN-GP | AppScanner | 70.2% | 71.8% | 46 |
| BFCN | 97.3% | 96.8% | 32 | |
| ATVITSC | 96.9% | 96.7% | 32 | |
| CMTSNN | 94.0% | 93.8% | 39 | |
| METAROCKETC | 96.6% | 96.2% | 37 | |
| TFE-GNN | 95.1% | 94.9% | 38 | |
| TFG-GNN | 95.8% | 95.7% | 32 | |
| HEDGAT | 98.1% | 98.1% | 27 |
表 2 各模型在3个数据集上的训练效率对比Table 2 Training efficiency comparison of different models on 3 datasets |
| 数据集 | 模型 | 单epoch训练时间/s | 全量数据训练 总耗时 /min |
| GenAI | AppScanner | 33 | 23 |
| BFCN | 35 | 23 | |
| ATVITSC | 39 | 25 | |
| CMTSNN | 57 | 38 | |
| METAROCKETC | 45 | 25 | |
| TFE-GNN | 38 | 23 | |
| TFG-GNN | 34 | 19 | |
| HEDGAT | 35 | 16 | |
| ICS- GenMal | AppScanner | 97 | 77 |
| BFCN | 84 | 49 | |
| ATVITSC | 88 | 45 | |
| CMTSNN | 96 | 66 | |
| METAROCKETC | 92 | 55 | |
| TFE-GNN | 96 | 62 | |
| TFG-GNN | 90 | 51 | |
| HEDGAT | 87 | 38 | |
| WGAN-GP | AppScanner | 67 | 51 |
| BFCN | 65 | 35 | |
| ATVITSC | 73 | 39 | |
| CMTSNN | 88 | 57 | |
| METAROCKETC | 69 | 43 | |
| TFE-GNN | 79 | 50 | |
| TFG-GNN | 74 | 39 | |
| HEDGAT | 70 | 32 |
表 3 各模型在不同场景下的性能结果Table 3 Performance results of different models on different scenarios |
| 模型名称 | 噪声干扰 准确率 | 噪声干 扰损失 | 数据不平 衡F1分数 |
| Base-GAT | 89.5% | 0.42 | 0.87 |
| Hetero-Subgraph | 90.7% | 0.36 | 0.89 |
| Distill-Single | 91.2% | 0.34 | 0.90 |
| Hetero+Distill-NoShare | 91.5% | 0.30 | 0.91 |
| Hetero+Shared | 92.6% | 0.25 | 0.92 |
| HEDGAT | 94.5% | 0.12 | 0.95 |
| 1 |
Scarselli F, Gori M, Tsoi A C, et al. The graph neural network model[J]. IEEE Transactions on Neural Networks, 2009, 20 (1): 61- 80.
|
| 2 |
Zhang C X, Song D J, Huang C, et al. Heterogeneous graph neural network[C]//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. New York: ACM, 2019: 793-803.
|
| 3 |
Liu Y, Wang X, Qu B, et al. ATVITSC: a novel encrypted traffic classification method based on deep learning[J]. IEEE Transactions on Information Forensics and Security, 2024, 19, 9374- 9389.
|
| 4 |
胡斌, 周志洪, 姚立红, 李建华等, 结合报文负载与流指纹特征的恶意流量检测[J]. 计算机工程, 2020, 46(11): 157-163.
Hu B, Zhou Z H, Yao L H, Li J H, et al. Malicious traffic detection combining packet payload and flow fingerprint features[J]. Computer Engineering, 2020, 46(11): 157-163.
|
| 5 |
Lu J Z, Zhang W S, Hamzei M, et al. The applications of machine learning mechanisms in the compositions of Internet of Things services: a systematic study, current progress, and future research agenda[J]. Engineering Applications of Artificial Intelligence, 2025, 147, 110345.
|
| 6 |
Zhu S Z, Xu X L, Gao H H, et al. CMTSNN: a deep learning model for multiclassification of abnormal and encrypted traffic of Internet of things[J]. IEEE Internet of Things Journal, 2023, 10 (13): 11773- 11791.
|
| 7 |
Li X H, Xu M F, Vijayakumar P, et al. Detection of low-frequency and multi-stage attacks in industrial Internet of things[J]. IEEE Transactions on Vehicular Technology, 2020, 69 (8): 8820- 8831.
|
| 8 |
Rey V, Sánchez P M, Huertas C A, et al. Federated learning for malware detection in IoT devices[J]. Computer Networks, 2022, 204, 108693.
|
| 9 |
Niktabe S, Lashkari A H, Roudsari A H. Unveiling DoH tunnel: toward generating a balanced DoH encrypted traffic dataset and profiling malicious behavior using inherently interpretable machine learning[J]. Peer-to-Peer Networking and Applications, 2024, 17 (1): 507- 531.
|
| 10 |
Lu J Z, Wang C L, Huang Y Y, et al. An adversarial example defense algorithm for intelligent driving[J]. IEEE Network, 2024, 38 (6): 98- 105.
|
| 11 |
程华, 谢金鑫, 陈立皇. 基于CNN的加密C&C通信流量识别方法[J]. 计算机工程, 2019, 45 (8): 31- 34,41.
Cheng H, Xie J X, Chen L H. CNN-based encrypted C & C communication traffic identification method[J]. Computer Engineering, 2019, 45 (8): 31- 34,41.
|
| 12 |
Kan H, Ren R. Research on data feature processing and detection of encrypted malicious traffic in graph neural networks[J]. International Journal of Network Security, 2024, 26 (6): 985- 991.
|
| 13 |
蒋彤彤, 尹魏昕, 蔡冰, 等. 基于层次时空特征与多头注意力的恶意加密流量识别[J]. 计算机工程, 2021, 47 (7): 101- 108.
Jiang T T, Yin W X, Cai B, et al. Encrypted malicious traffic identification based on hierarchical spatiotemporal feature and multi-head attention[J]. Computer Engineering, 2021, 47 (7): 101- 108.
|
| 14 |
Zhang H Z, Yue H D, Xiao X, et al. Revolutionizing encrypted traffic classification with MH-net: a multi-view heterogeneous graph model[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2025, 39 (1): 1048- 1056.
|
| 15 |
Zhao J J, Cui Z W, Fu J S, et al. A unified framework for robust encrypted malicious traffic detection in adverse environments via graph structure learning[J]. IEEE Transactions on Network Science and Engineering, 2026, 13, 245- 261.
|
| 16 |
Zha C, Wang Z Y, Fan Y F, et al. DM-IDS: a network intrusion detection method based on dual-modal fusion[J]. IEEE Transactions on Network and Service Management, 2025, 22 (4): 3646- 3661.
|
| 17 |
Wang M L, Zhang B W, Zang X D, et al. Malicious traffic classification via edge intelligence in IIoT[J]. Mathematics, 2023, 11 (18): 3951.
|
| 18 |
Ullah S, Boulila W, Koubâa A, et al. MAGRU-IDS: a multi-head attention-based gated recurrent unit for intrusion detection in IIoT networks[J]. IEEE Access, 2023, 11, 114590- 114601.
|
| 19 |
Bovenzi G, Ciuonzo D, Krolikowski J, et al. Lightweight GenAI for network traffic synthesis: fidelity, augmentation, and classification[PP]. arXiv preprint arXiv: 2603. 25507, 2026.
|
| 20 |
Aceto G, Giampaolo F, Guida C, et al. Synthetic and privacy-preserving traffic trace generation using generative AI models for training network intrusion detection systems[J]. Journal of Network and Computer Applications, 2024, 229, 103926.
|
| 21 |
Taylor V F, Spolaor R, Conti M, et al. Robust smartphone app identification via encrypted network traffic analysis[J]. IEEE Transactions on Information Forensics and Security, 2018, 13 (1): 63- 78.
|
/
| 〈 |
|
〉 |