计算机科学

计算机科学 ›› 2024, Vol. 51 ›› Issue (11): 379-388.doi: 10.11896/jsjkx.231200034

• 信息安全 • 上一篇    下一篇

参数解耦在差分隐私保护下的联邦学习中的应用

王梓行1, 杨敏1, 魏子重2   

  1. 1 空天信息安全与可信计算教育部重点实验室(武汉大学国家网络安全学院) 武汉 430072
    2 浪潮集团科学研究院 济南 250101
  • 收稿日期:2023-12-05 修回日期:2024-04-02 出版日期:2024-11-15 发布日期:2024-11-06
  • 通讯作者: 杨敏(yangm@whu.edu.cn)
  • 作者简介:(wzihang@whu.edu.cn)
  • 基金资助:
    国家自然科学基金(62172308);国家重点基础研究发展计划(2021YFB2700200)

Application of Parameter Decoupling in Differentially Privacy Protection Federated Learning

WANG Zihang1, YANG Min1, WEI Zichong2   

  1. 1 Key Laboratory of Aerospace Information Security and Trusted Computing,Ministry of Education,School of Cyber Science and Engineering,Wuhan University,Wuhan 430072,China
    2 Inspur Group Scientific Research Institute,Jinan 250101,China
  • Received:2023-12-05 Revised:2024-04-02 Online:2024-11-15 Published:2024-11-06
  • About author:WANG Zihang,born in 1999,postgra-duate,is a member of CCF(No.R6802G).His main research interests include differential privacy and federated learning.
    YANG Min,born in 1975,Ph.D,asso-ciate professor,master supervisor,is a member of CCF(No.51131M).Her main research interests include information security and applied cryptography.
  • Supported by:
    National Natural Science Foundation of China(62172308) and National Basic Research Program of China(2021YFB2700200).

PDF (PC)

248

摘要： 联邦学习(Federated Learning,FL)是一种先进的隐私保护机器学习技术,其通过多方协作,在无需集中聚合原始数据的情况下,交换模型参数以训练共享模型。尽管在FL中参与方不需要显式地共享数据,但许多研究表明,其仍然面临多种隐私推理攻击,从而导致隐私信息泄露。为应对这一问题,学术界提出了多种解决方案。其中,一种严格保障隐私的方法是将本地化差分隐私(Local Differential Privacy,LDP)技术应用于联邦学习。该技术在参与方上传模型参数前对其添加随机噪声,能有效地抵御恶意攻击者的推理攻击。然而,LDP引入的噪声会造成模型性能下降。同时,最新研究指出,这种性能下降与LDP在客户端之间引入了额外的异构性有关。针对LDP使得FL性能下降的问题,提出了差分隐私保护下基于参数解耦的联邦学习方案(PD-LDPFL):除了服务器下发的基础模型外,每个客户端在本地还额外学习了个性化输入和输出模型。该方案在客户端传输时仅上传添加噪声后的基础模型的参数,而个性化模型的参数被保留在本地,自适应改变客户端本地数据的输入和输出分布,缓解LDP引入的额外异构性以减少精度损失。此外,研究发现,即使在采用较高的隐私预算的情况下,该方案也能天然地抵御一些基于梯度的隐私推理攻击,如深度梯度泄露等攻击方法。在MNIST,FMNIST和CIFAR-10这3个常用数据集上进行了实验,结果表明:相比传统的差分隐私联邦学习方法,该方案不仅可以获得更好的性能,而且还提供了额外的安全性。

关键词: 联邦学习, 差分隐私, 异构性, 参数解耦, 隐私保护

Abstract: Federated learning(FL) is an advanced privacy preserving machine learning technique that exchanges model parameters to train shared models through multi-party collaboration without the need for centralized aggregation of raw data.Although participants in FL do not need to explicitly share data,many studies show that they still face various privacy inference attacks,leading to privacy information leakage.To address this issue,the academic community has proposed various solutions.One of the strict privacy protection methods is to apply Local differential privacy(LDP) technology to federated learning.This technology adds random noise to the model parameters before they are uploaded by participants,to effectively resist inference attacks from malicious attackers.However,the noise introduced by LDP can reduce the model performance.Meanwhile,the latest research suggests that this performance decline is related to the additional heterogeneity introduced by LDP between clients.A parameter decoupling based federated learning scheme(PD-LDPFL) with differential privacy protection is proposed to address the issue of FL performance degradation caused by LDP.In addition to the basic model issued by the server,each client also learns personalized input and output models locally.This scheme only uploads the parameters of the basic model with added noise during client transmission,while the personalized model is retained locally,adaptively changing the input and output distribution of the client’s local data to alleviate the additional heterogeneity introduced by LDP and reduce accuracy loss.In addition,research has found that even with a higher privacy budget,this scheme can naturally resist some gradient based privacy inference attacks,such as deep gradient leakage and other attack methods.Through experiments on three commonly used datasets,MNIST,FMNIST,and CIFAR-10,the results show that this scheme not only achieves better performance compared to traditional differential privacy federated learning,but also provides additional security.

Key words: Federated learning, Differential privacy, Heterogeneity, Parameter decoupling, Privacy preserving

中图分类号: 

  • TP309
[1] TANG P,XU H M,MA C.ProtoTransfer:Cross-Modal Prototype Transfer for Point Cloud Segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.2023:3337-3347.
[2] ZHAN F,YU Y,WU R,et al.Marginal contrastive correspondence for guided image generation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2022:10663-10672.
[3] LEE P,BUBECK S,PETRO J.Benefits,limits,and risks ofGPT-4 as an AI chatbot for medicine[J].New England Journal of Medicine,2023,388(13):1233-1239.
[4] MCMAHAN B,MOORE E,RAMAGE D,et al.Communica-tion-efficient learning of deep networks from decentralized data[C]//Artificial Intelligence and Statistics.PMLR,2017:1273-1282.
[5] PHONG L T,AONO Y,HATASHI T,et al.Privacy-preserving deep learning:Revisited and enhanced[C]//Applications and Techniques in Information Security:8th International Confe-rence.2017:100-110.
[6] WEI W,LIU L,LOPER M,et al.A framework for evaluatinggradient leakage attacks in federated learning[J].ESORICS 2020:25th European Symposium on Research in Computer Security,2020,12308:545-566.
[7] AONO Y,HAYASHI T,WANG L,et al.Privacy-preservingdeep learning via additively homomorphic encryption[J].IEEE Transactions on Information Forensics and Security,2017,13(5):1333-1345.
[8] YIN H,MALLYA A,VAHDAT A,et al.See through gra-dients:Image batch recovery via gradinversion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021:16337-16346.
[9] GEIPING J,BAUERMEISTER H,DRÖGE H,et al.Inverting gradients-how easy is it to break privacy in federated learning?[J].Advances in Neural Information Processing Systems,2020,33:16937-16947.
[10] MOTHUKURI V,PARIZI R M,POURIYEH S,et al.A surveyon security and privacy of federated learning[J].Future Generation Computer Systems,2021,115:619-640.
[11] DWORK C,MCSHERRY F,NISSIM K,et al.Calibrating noise to sensitivity in private data analysis[C]//Thirty-Ninth ACM Symposium on Theory of Computing ACM.2007:75-84.
[12] DWORK C,ROTH A.The algorithmic foundations of differential privacy[J].Foundations and Trends in Theoretical Compu-ter Science,2014,9(3/4):211-407.
[13] KASIVISWANATHAN S P,LEE H K,NISSIM K,et al.What can we learn privately?[J].SIAM Journal on Computing,2011,40(3):793-826.
[14] KIM M,JAIN A K,LIU X.Adaface:Quality adaptive margin for face recognition[C]//Proceedings of the IEEE/CVF Confe-rence on Computer Vision and Pattern Recognition.2022:18750-18759.
[15] XU J,HE X,LI H.Deep learning for matching in search and recommendation[C]//The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval.2018:1365-1368.
[16] WANG T,HU X,LIU Z,et al.Sparse2Dense:Learning to densify 3d features for 3d object detection[J].Advances in Neural Information Processing Systems,2022,35:38533-38545.
[17] YE M,FANG X,DU B,et al.Heterogeneous federated lear-ning:State-of-the-art and research challenges[J].ACM Computing Surveys,2023,56(3):1-44.
[18] FU J,CHEN Z,HAN X.Adap DP-FL:Differentially PrivateFederated Learning with Adaptive Noise[C]//2022 IEEE International Conference on Trust,Security and Privacy in Computing and Communications(TrustCom).IEEE,2022:656-663.
[19] HUANG X,DING Y,JIANG Z L,et al.DP-FL:a novel diffe-rentially private federated learning framework for the unbalanced data[J].World Wide Web,2020,23:2529-2545.
[20] YANG Y,HUI B,YUAN H,et al.PrivateFL:Accurate,Diffe-rentially Private Federated Learning via Personalized Data Transformation[C]//32nd USENIX Security Symposium(USENIX Security 23).2023:1595-1612.
[21] LI Q,WEN Z,WU Z,et al.A survey on federated learning systems:Vision,hype and reality for data privacy and protection[J].IEEE Transactions on Knowledge and Data Engineering,2021,35(4):3347-3366.
[22] KAIROUZ P,MCMAHAN H B,AVENT B,et al.Advancesand open problems in federated learning[J].Foundations and Trends in Machine Learning,2021,14(1/2):1-210.
[23] WEI K,LI J,DING M,et al.Federated learning with differential privacy:Algorithms and performance analysis[J].IEEE Tran-sactions on Information Forensics and Security,2020,15:3454-3469.
[24] SAJADMANESH S,SHAMSABADI A S,BELLET A,et al.Gap:Differentially private graph neural networks with aggregation perturbation[C]//32nd USENIX Security Symposium.2023.
[25] SUN L,QIAN J,CHEN X.LDP-FL:Practical Private Aggregation in Federated Learning with Local Differential Privacy[C]//Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence.International Joint Conferences on Artificial Intelligence Organization,2021.
[26] WANG N,XIAO X,YANG Y,et al.Collecting and analyzingmultidimensional data with local differential privacy[C]//2019 IEEE 35th International Conference on Data Engineering(ICDE).IEEE,2019:638-649.
[27] DUCHI J C,JORDAN M I,WAINWRIGHT M J.Minimax optimal procedures for locally private estimation[J].Journal of the American Statistical Association,2018,113(521):182-201.
[28] BU Z,DONG J,LONG Q,et al.Deep Learning with Gaussian Differential Privacy[J].Harvard Data Science Review,2020.
[29] ABADI M,CHU A,GOODFELLOW I,et al.Deep learning with differential privacy[C]//Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security.2016:308-318.
[30] WANG N,XIAO X,YANG Y,et al.Collecting and analyzingmultidimensional data with local differential privacy[C]//2019 IEEE 35th International Conference on Data Engineering(ICDE).IEEE,2019:638-649.
[31] ZHU L,LIU Z,HAN S.Deep leakage from gradients[C]//Advances in Neural Information Processing Systems.2019:14747-14756.
[32] KARIMIREDDY S P,KALE S,MOHRI M,et al.Scaffold:Stochastic controlled averaging for federated learning[C]//International Conference on Machine Learning.PMLR,2020:5132-5143.
[33] TAN Y,LONG G,LIU L,et al.Fedproto:Federated prototype learning across heterogeneous clients[C]//Proceedings of the AAAI Conference on Artificial Intelligence.2022:8432-8440.
[34] LI Q,DIAO Y,CHEN Q,et al.Federated learning on non-iid data silos:An experimental study[C]//2022 IEEE 38th International Conference on Data Engineering(ICDE).IEEE,2022:965-978.
[35] SATTLER F,MÜLLER K R,SAMEK W.Clustered federated learning:Model-agnostic distributed multitask optimization under privacy constraints[J].IEEE Transactions on Neural Networks and Learning Systems,2020,32(8):3710-3722.
[36] WU Q,HE K,CHEN X.Personalized federated learning for intelligent IoT applications:A cloud-edge based framework[J].IEEE Open Journal of the Computer Society,2020,1:35-44.
[37] HÖNIG R,ZHAO Y,MULLINS R.DAdaQuant:Doubly-adap-tive quantization for communication-efficient Federated Learning[C]//International Conference on Machine Learning.PMLR,2022:8852-8866.
[38] WANG Y,LIN L,CHEN J.Communication-efficient adaptivefederated learning[C]//International Conference on Machine Learning.PMLR,2022:22802-22838.
[39] LI T,SAHU A K,TALWALKAR A,et al.Federated learning:Challenges,methods,and future directions[J].IEEE Signal Processing Magazine,2020,37(3):50-60.
[40] NIELSEN M A.Neural networks and deep learning[M].SanFrancisco,CA,USA:Determination Press,2015.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
地址：重庆市渝北区洪湖西路18号    邮编：401121  
电话：023-63500828（编辑部） 023-67039612（会议/专题） 023-67039623（财务/发票）
E-mail：jsjkx12@163.com
本系统由北京玛格泰克科技发展有限公司设计开发