异构数据环境下的联邦学习综述
收稿日期: 2025-03-18
网络出版日期: 2025-07-18
基金资助
国家重点研发计划青年科学家项目（2023YFB2704000）
版权
A survey on federated learning in heterogeneous data environments
Received date: 2025-03-18
Online published: 2025-07-18
Copyright
汤尧 , 刘健 . 异构数据环境下的联邦学习综述[J]. 网络空间安全科学学报, 2025 , 3(2) : 2 -11 . DOI: 10.20172/j.issn.2097-3136.250201
With the continuous development of multimodal and large language models, the heterogeneity of data involved in artificial intelligence (AI) training is gradually increasing. This has brought new challenges and opportunities to federated learning (FL). Relevant studies of federated learning in heterogeneous data scenarios were reviewed, and the indicators and methods for assessing data shift were summarized from the data and model levels with the existing federated learning solutions reviewed from multiple technical routes such as the multi-task learning and data distillation. Finally, the integration directions of federated learning with emerging technologies such as large models and multimodality were explored, and the future development trends were predicted.
Key words： federated learning; data privacy; heterogeneous data
表 1 现有技术的核心思想Table 1 Core idea of the existing technologies |
| 方案 | 核心思想 |
| 原型学习 | 通过交换代表性样本（原型）传递数据信息 |
| 知识蒸馏 | 将教师模型的知识迁移到学生模型 |
| 多任务学习 | 同时训练多个相关任务以提高泛化能力 |
| 对比学习 | 通过比较学习正负样本对来增强数据表示相似性 |
| 数据增强 | 通过数据生成增加训练数据的多样性， 以缓解本地数据不平衡问题 |
| 预训练模型 | 使用预训练模型减少训练时间和提高模型准确性 |
| 1 |
MCMAHAN B,MOORE E,RAMAGE D,et al. Communication-efficient learning of deep networks from decentralized data[C]//Artificial Intelligence and Statistics. PMLR,2017:1273-1282.
|
| 2 |
YUROCHKIN M,AGARWAL M,GHOSH S,et al. Bayesian nonparametric federated learning of neural networks[C]//International conference on machine learning. PMLR,2019:7252-7261.
|
| 3 |
CHEN X, CHEN T, SUN H, et al. Distributed training with heterogeneous data: Bridging median-and mean-based algorithms[J]. Advances in Neural Information Processing Systems, 2020, 33, 21616- 21626.
|
| 4 |
LI X, HUANG K, YANG W, et al. On the convergence of FedAvg on Non-IID data[J]. arXiv preprint, arXiv:, 1907, 02189, 2019.
|
| 5 |
CHEN C, FENG X, ZHOU J, et al. Federated large language model: A position paper[J]. arXiv preprint, arXiv:, 2307, 08925, 2023.
|
| 6 |
CHE L, WANG J, ZHOU Y, et al. Multimodal federated learning: A Survey[J]. Sensors, 2023, 23 (15): 6986.
|
| 7 |
AAMER B, CHERGUI H, BENJILLALI M, et al. Entropy-driven stochastic federated learning in Non-IID 6G edge-ran[J]. Frontiers in Communications and Networks, 2021, 2, 739414.
|
| 8 |
YANG H, FANG M, LIU J. Achieving linear speedup with partial worker participation in Non-IID federated learning[J]. arXiv preprint, arXiv:, 2101, 11203, 2021.
|
| 9 |
ZAWAD S,ALI A,CHEN P Y,et al. Curse or redemption? How data heterogeneity affects the robustness of federated learning[C]//Proceedings of the AAAI conference on artificial intelligence. 2021,35(12):10807-10814.
|
| 10 |
PARK C,CHOI T,KIM T,et al. FedGeo:Privacy-preserving user next location prediction with federated learning[C]//Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems. ACM,2023:1-10.
|
| 11 |
ORTIGOSA-HERNÁNDEZ J, INZA I, LOZANO J A. Measuring the class-imbalance extent of multi-class problems[J]. Pattern Recognition Letters, 2017, 98, 32- 38.
|
| 12 |
QU L,ZHOU Y,LIANG P P,et al. Rethinking architecture design for tackling data heterogeneity in federated learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE,2022:10061-10071.
|
| 13 |
LI A, SUN J, WANG B, et al. LotteryFL: Personalized and communication-efficient federated learning with lottery ticket hypothesis on Non-IID datasets[J]. arXiv preprint, arXiv: , 2008, 03371, 2020.
|
| 14 |
JIMENEZ G D M, ANAGNOSTOPOULOS A, CHATZIGIANNAKIS I, et al. FedArtML: A tool to facilitate the generation of Non-IID datasets in a controlled way to support federated learning research[J]. IEEE Access, 2024, 12, 2169- 3536.
|
| 15 |
TAN Q,WU S,TAO Y. Privacy-enhanced federated learning for Non-IID data[J]. Mathematics,2023,11(19):4123.
|
| 16 |
XU Y,LI Y,LUO H,et al. FBLG:A local graph based approach for handling dual skewed Non-IID data in federated learning[C]//Proceedings of the 33rd International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization. 2024,8:5289-5297.
|
| 17 |
ZHANG L, GAO G, ZHANG H. Spatial-temporal federated learning for lifelong person re-identification on distributed edges[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2025, 35 (2): 1884- 1896.
|
| 18 |
SHU J, YANG T, LIAO X, et al. Clustered federated multitask learning on Non-IID data with enhanced privacy[J]. IEEE Internet of Things Journal, 2022, 10 (4): 3453- 3467.
|
| 19 |
LUO G, CHEN N, HE J, et al. Privacy-preserving clustering federated learning for Non-IID data[J]. Future Generation Computer Systems, 2024, 154, 384- 395.
|
| 20 |
HALLER M,LENZ C,NACHTIGALL R,et al. Handling Non-IID data in federated learning:An experimental evaluation towards unified metrics[C]//2023 IEEE International Conference on Dependable,Autonomic and Secure Computing,International Conference on Pervasive Intelligence and Computing,International Conference on Cloud and Big Data Computing,International Conference on Cyber Science and Technology Congress. IEEE,2023:762-770.
|
| 21 |
CHEN A, FU Y, SHA Z, et al. An EMD-based adaptive client selection algorithm for federated learning in heterogeneous data scenarios[J]. Frontiers in Plant Science, 2022, 13, 908814.
|
| 22 |
ZHAO H. Non-IID quantum federated learning with one-shot communication complexity[J]. Quantum Machine Intelligence, 2023, 5 (1): 3.
|
| 23 |
HSIEH K,PHANISHAYEE A,MUTLU O,et al. The Non-IID data quagmire of decentralized machine learning[C]//International Conference on Machine Learning. PMLR,2020:4387-4398.
|
| 24 |
TAN Y, LONG G, MA J, et al. Federated learning from pre-trained models: A contrastive learning approach[J]. Advances in Neural Information Processing Systems, 2022, 35, 19332- 19344.
|
| 25 |
LI G, WU W, SUN Y, et al. Visual prompt based personalized federated learning[J]. arXiv preprint, arXiv:, 2303, 08678, 2023.
|
| 26 |
TAN Y,LONG G,LIU L,et al. FedProto:Federated prototype learning across heterogeneous clients[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2022,36(8):8432-8440.
|
| 27 |
WEN H, WU Y, HU J, et al. Communication-efficient federated learning on Non-IID data using two-step knowledge distillation[J]. IEEE Internet of Things Journal, 2023, 10 (19): 17307- 17322.
|
| 28 |
HAN S,PARK S,WU F,et al. FedX:Unsupervised federated learning with cross knowledge distillation[C]//European Conference on Computer Vision. Cham:Springer Nature Switzerland,2022:691-707.
|
| 29 |
ZHUANG W,WEN Y,LYU L,et al. MAS:Towards resource-efficient federated multiple-task learning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE,2023:23414-23424.
|
| 30 |
LI Q,HE B,SONG D. Model-contrastive federated learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE,2021:10713-10722.
|
| 31 |
LI Z,SHAO J,MAO Y,et al. Federated learning with GAN-based data synthesis for Non-IID clients[C]//International Workshop on Trustworthy Federated Learning. Cham:Springer International Publishing,2022:17-32.
|
| 32 |
GUO W,YAO Z,LIU Y,et al. A new federated learning model for host intrusion detection system under Non-IID data[C]//2023 IEEE International Conference on Systems,Man,and Cybernetics (SMC). IEEE,2023:494-500.
|
| 33 |
AVELINO J G, CAVALCANTI G D, CRUZ R M. Resampling strategies for imbalanced regression: A survey and empirical analysis[J]. Artificial Intelligence Review, 2024, 57(4): 82.
|
| 34 |
KANG Y, FAN T, GU H, et al. Grounding foundation models through federated transfer learning: A general framework[J]. arXiv preprint, arXiv:, 2311, 17431, 2023.
|
| 35 |
LI T, SAHU A K, ZAHEER M, et al. Federated optimization in heterogeneous networks[J]. Proceedings of Machine Learning and Systems, 2020, 2, 429- 450.
|
| 36 |
KARIMIREDDY S P,KALE S,MOHRI M,et al. SCAFFOLD:Stochastic controlled averaging for federated learning[C]//International Conference on Machine Learning. PMLR,2020:5132-5143.
|
| 37 |
WANG J, LIU Q, LIANG H, et al. Tackling the objective inconsistency problem in heterogeneous federated optimization[J]. Advances in Neural Information Processing Systems, 2020, 33, 7611- 7623.
|
| 38 |
ZHANG X,LI Y,LI W,et al. Personalized federated learning via variational Bayesian inference[C]//International Conference on Machine Learning. PMLR,2022:26293-26310.
|
| 39 |
TANG Z,ZHANG Y,SHI S,et al. Virtual homogeneity learning:Defending against data heterogeneity in federated learning[C]//International Conference on Machine Learning. PMLR,2022:21111-21132.
|
| 40 |
YI L, YU H, WANG G, et al. pFedLoRA: Model-heterogeneous personalized federated learning with LoRA tuning[J]. arXiv preprint, arXiv:, 2310, 13283, 2023.
|
| 41 |
SUN Y, LI Z, LI Y, et al. Improving LoRA in privacy-preserving federated learning[J]. arXiv preprint, arXiv:, 2403, 12313, 2024.
|
| 42 |
BABAKNIYA S, ELKORDY A R, EZZELDIN Y H, et al. SLoRA: Federated parameter efficient fine-tuning of language models[J]. arXiv preprint, arXiv:, 2308, 06522, 2023.
|
| 43 |
CHE T, LIU J, ZHOU Y, et al. Federated learning of large language models with parameter-efficient prompt tuning and adaptive optimization[J]. arXiv preprint, arXiv:, 2310, 15080, 2023.
|
| 44 |
LIN Z, SUN Y, SHI Y, et al. Efficient federated prompt tuning for black-box large pre-trained models[J]. arXiv preprint, arXiv:, 2310, 03123, 2023.
|
| 45 |
ZHANG X, LI M, CHANG X, et al. FedYolo: Augmenting federated learning with pretrained transformers[J]. arXiv preprint, arXiv:, 2307, 04905, 2023.
|
| 46 |
BIAN J, WANG L, XU J. Prioritizing modalities: Flexible importance scheduling in federated multimodal learning[J]. arXiv preprint, arXiv:, 2408, 06549, 2024.
|
| 47 |
XIONG B, YANG X, QI F, et al. A unified framework for multi-modal federated learning[J]. Neurocomputing, 2022, 480, 110- 118.
|
| 48 |
CHEN H,ZHANG Y,KROMPASS D,et al. FedDAT:An approach for foundation model finetuning in multi-modal heterogeneous federated learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024,38(10):11285-11293.
|
| 49 |
CHO Y J, MANOEL A, JOSHI G, et al. Heterogeneous ensemble knowledge transfer for training large models in federated learning[J]. arXiv preprint, arXiv: , 2204, 12703, 2022.
|
| 50 |
WANG H,ZHAO H,WANG Y,et al. FedKC:Federated knowledge composition for multilingual natural language understanding[C]//Proceedings of the ACM Web Conference 2022. 2022:1839-1850.
|
| 51 |
YU Q, LIU Y, WANG Y, et al. Multimodal federated learning via contrastive representation ensemble[J]. arXiv preprint, arXiv:, 2302, 08888, 2023.
|
| 52 |
WEI G, WANG F, SHAH A, et al. Dual prompt tuning for domain-aware federated learning[J]. arXiv preprint, arXiv:, 2310, 03103, 2023.
|
| 53 |
WANG R,YU T,WU J,et al. Federated domain adaptation for named entity recognition via distilling with heterogeneous tag sets[C]//Findings of the Association for Computational Linguistics:ACL 2023. 2023:7449-7463.
|
| 54 |
ZHANG Z,QI F,XU C. Enhancing storage and computational efficiency in federated multimodal learning for large-scale models[C]//41st International Conference on Machine Learning. PMLR,2024:59685-59699.
|
/
| 〈 |
|
〉 |
