计算机科学 ›› 2022, Vol. 49 ›› Issue (9): 183-193.doi: 10.11896/jsjkx.220500263
鲁晨阳, 邓苏, 马武彬, 吴亚辉, 周浩浩
LU Chen-yang, DENG Su, MA Wu-bin, WU Ya-hui, ZHOU Hao-hao
摘要： 联邦学习是一种新的面向隐私保护的分布式学习范式,相比传统分布式机器学习方法,其特点为各客户端通信、设备算力和存储能力存在较大差异(设备异构),各客户端数据分布和数量存在较大差异(数据异构)以及高通信消耗等。在客户端异构条件(包括设备异构和数据异构)下,客户端的数据分布区别较大,导致模型收敛速度显著降低,特别是在极端的数据异构情况下,传统的联邦学习算法无法收敛,并且训练曲线随着本地迭代轮次的增加出现大幅的波动。针对联邦学习中,客户端异构给模型训练带来的影响,提出了利用分层抽样优化的联邦学习算法——FedSSO。FedSSO使用了基于密度的聚类方法将总体客户端划入不同的聚类中,使得每个聚类中的客户端具有较高的相似度,再按样本权重从不同聚类中抽取可用客户端参与训练,因此所有种类的数据都会按样本权重参与每轮训练,使模型加速收敛到全局最优解;同时,设定了学习率递减和本地迭代轮次选择机制,以保证模型的收敛性。从理论和实验中证明了FedSSO的收敛性,并且在公开数据集MNIST,Cifar-10和Sentiment140上与其他联邦学习算法进行了对比,实验结果证明FedSSO的训练效果更优。
中图分类号:
| [1]LI T,SAHU A K,TALWALKAR A,et al.Federated Learning:Challenges,Methods,and Future Directions[J].IEEE Signal Processing Magazine,2020,37(3):50-60. [2]MCMAHAN B,MOORE E,RAMAGE D,et al.Communica-tionefficient learning of deep networks from decentralized data[C]//Artificial Intelligence and Statistics.PMLR,2017:1273-1282. [3]MCMAHAN H B,RAMAGE D,TALWAR K,et al.Learning Differentially Private Recurrent Language Models [J].arXiv:1710.06963,2017. [4]YANG Q,LIU Y,CHEN T,et al.Federated Machine Learning:Concept and Applications[J].ACM Transactions on Intelligent Systems and Technology,2019,10(2):1-19. [5]HSIEH K,PHANISHAYEE A,MUTLU O,et al.The Non-IID Data Quagmire of Decentralized Machine Learning [J].arXiv:1910.00189,2020. [6]LI T,SAHU A K,ZAHEER M,et al.Federated Optimization in Heterogeneous Networks[J].arXiv.1812.06127,2018. [7]HARD A,RAO K,MATHEWS R,et al.Federated Learning for Mobile Keyboard Prediction [J].arXiv.1811.03604,2018. [8]YANG T,ANDREW G,EICHNER H,et al.Applied Federated Learning:Improving Google Keyboard Query Suggestions [J].arXiv.1812.02903,2018. [9]BOYD S,PARIKH N,CHU E,et al.Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers[J].Foundations & Trends in Machine Learning,2010,3(1):1-122. [10]DEKEL O,GILAD-BACHRACH R,SHAMIR O,et al.Optimal Distributed Online Prediction Using Mini-Batches [J].Journal of Machine Learning Research,2012,13(1):165-202. [11]RICHTÁRIK P,TAKÁČ M.Distributed Coordinate DescentMethod for Learning with Big Data [J].Journal of Machine Learning Research,2016,17(75):1-25. [12]ZHANG S,CHOROMANSKA A,LECUN Y.Deep learningwith Elastic Averaging SGD [J].arXiv.1412.6651,2014. [13]BONAWITZ K,IVANOV V,KREUTER B,et al.Practical Secure Aggregation for Privacy-Preserving Machine Learning[C]//The 2017 ACM SIGSAC Conference.ACM,2017:1175-1191. [14]BONAWITZ K,EICHNER H,GRIESKAMP W,et al.Towards Federated Learning at Scale:System Design [J].arXiv.1902.01046,2019. [15]MOHRI M,SIVEK G,SURESH A T.Agnostic FederatedLearning [C]//International Conference on Machine Learning.PMLR,2019. [16]HU H,WANG D,WU C.Distributed Machine Learning th-rough Heterogeneous Edge Systems[C]//AAAI Conference on Artificial Intelligence.2020:7179-7186. [17]PETER K,BRENDAN H,MCMAHAN H B,et al.Advancesand Open Problems in Federated Learning [J].arXiv.1912.04977,2019. [18]ZHAO Y,LI M,LAI L,et al.Federated Learning with Non-IID Data [J].arXiv.1806.00582,2018. [19]GHOSH A,CHUNG J,DONG Y,et al.An Efficient Framework for Clustered Federated Learning [J].arXiv:2006.04088,2020. [20]SATTLER F,KR MÜLLER,SAMEK W.Clustered Federated Learning:Model-Agnostic Distributed Multitask Optimization Under Privacy Constraints [J].IEEE Trans Neural Netw Learn Syst,2021,32(8):3710-3722. [21]YAN Y,NIU C,DING Y,et al.Distributed Non-Convex Optimization with Sublinear Speedup under Intermittent Client Availability [J].arXiv.2002.07399,2020. [22]ANKERST M,BREUNIG M M,KRIEGEL H P,et al.OP-TICS:ordering points to identify the clustering structure [J].SIGMOD Record:Special Interest Group on Management Data,1999,28(2):49-60. [23]LI X,HUANG K,YANG W,et al.On the Convergence of FedAvg on Non-IID Data [J].arXiv:1907.02189,2020. [24]LECUN Y,BOTTOU L.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE,1998,86(11):2278-2324. |
| [1] | 柴慧敏, 张勇, 方敏. 基于特征相似度聚类的空中目标分群方法 Aerial Target Grouping Method Based on Feature Similarity Clustering 计算机科学, 2022, 49(9): 70-75. https://doi.org/10.11896/jsjkx.210800203 |
| [2] | 汤凌韬, 王迪, 张鲁飞, 刘盛云. 基于安全多方计算和差分隐私的联邦学习方案 Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy 计算机科学, 2022, 49(9): 297-305. https://doi.org/10.11896/jsjkx.210800108 |
| [3] | 吕由, 吴文渊. 隐私保护线性回归方案与应用 Privacy-preserving Linear Regression Scheme and Its Application 计算机科学, 2022, 49(9): 318-325. https://doi.org/10.11896/jsjkx.220300190 |
| [4] | 陈明鑫, 张钧波, 李天瑞. 联邦学习攻防研究综述 Survey on Attacks and Defenses in Federated Learning 计算机科学, 2022, 49(7): 310-323. https://doi.org/10.11896/jsjkx.211000079 |
| [5] | 鲁晨阳, 邓苏, 马武彬, 吴亚辉, 周浩浩. 基于DBSCAN聚类的集群联邦学习方法 Clustered Federated Learning Methods Based on DBSCAN Clustering 计算机科学, 2022, 49(6A): 232-237. https://doi.org/10.11896/jsjkx.211100059 |
| [6] | 郁舒昊, 周辉, 叶春杨, 王太正. SDFA:基于多特征融合的船舶轨迹聚类方法研究 SDFA:Study on Ship Trajectory Clustering Method Based on Multi-feature Fusion 计算机科学, 2022, 49(6A): 256-260. https://doi.org/10.11896/jsjkx.211100253 |
| [7] | 毛森林, 夏镇, 耿新宇, 陈剑辉, 蒋宏霞. 基于密度敏感距离和模糊划分的改进FCM算法 FCM Algorithm Based on Density Sensitive Distance and Fuzzy Partition 计算机科学, 2022, 49(6A): 285-290. https://doi.org/10.11896/jsjkx.210700042 |
| [8] | 陈景年. 一种适于多分类问题的支持向量机加速方法 Acceleration of SVM for Multi-class Classification 计算机科学, 2022, 49(6A): 297-300. https://doi.org/10.11896/jsjkx.210400149 |
| [9] | 闫萌, 林英, 聂志深, 曹一凡, 皮欢, 张兰. 一种提高联邦学习模型鲁棒性的训练方法 Training Method to Improve Robustness of Federated Learning 计算机科学, 2022, 49(6A): 496-501. https://doi.org/10.11896/jsjkx.210400298 |
| [10] | 刘丽, 李仁发. 医疗CPS协作网络控制策略优化 Control Strategy Optimization of Medical CPS Cooperative Network 计算机科学, 2022, 49(6A): 39-43. https://doi.org/10.11896/jsjkx.210300230 |
| [11] | 王健. 基于隐私保护的反向传播神经网络学习算法 Back-propagation Neural Network Learning Algorithm Based on Privacy Preserving 计算机科学, 2022, 49(6A): 575-580. https://doi.org/10.11896/jsjkx.211100155 |
| [12] | 陈佳舟, 赵熠波, 徐阳辉, 马骥, 金灵枫, 秦绪佳. 三维城市场景中的小物体检测 Small Object Detection in 3D Urban Scenes 计算机科学, 2022, 49(6): 238-244. https://doi.org/10.11896/jsjkx.210400174 |
| [13] | 邢云冰, 龙广玉, 胡春雨, 忽丽莎. 基于SVM的类别增量人体活动识别方法 Human Activity Recognition Method Based on Class Increment SVM 计算机科学, 2022, 49(5): 78-83. https://doi.org/10.11896/jsjkx.210400024 |
| [14] | 朱哲清, 耿海军, 钱宇华. 面向化学结构的线段聚类算法 Line-Segment Clustering Algorithm for Chemical Structure 计算机科学, 2022, 49(5): 113-119. https://doi.org/10.11896/jsjkx.210700131 |
| [15] | 张宇姣, 黄锐, 张福泉, 隋栋, 张虎. 基于菌群优化的近邻传播聚类算法研究 Study on Affinity Propagation Clustering Algorithm Based on Bacterial Flora Optimization 计算机科学, 2022, 49(5): 165-169. https://doi.org/10.11896/jsjkx.210800218 |
|
||
首页