Computer Science

Computer Science ›› 2022, Vol. 49 ›› Issue (7): 310-323.doi: 10.11896/jsjkx.211000079

• Information Security • Previous Articles     Next Articles

Survey on Attacks and Defenses in Federated Learning

CHEN Ming-xin1,2, ZHANG Jun-bo1,2, LI Tian-rui1   

  1. 1 School of Computing and Artificial Intelligence,Southwest Jiaotong University,Chengdu 611756,China
    2 JD iCity,JD Technology,Beijing 100098,China
  • Received:2021-10-12 Revised:2022-01-29 Online:2022-07-15 Published:2022-07-12
  • About author:CHEN Ming-xin,born in 1997,postgraduate,is a member of China Computer Federation.His main research interests include federated learning and so on.
    ZHANG Jun-bo,born in 1986,Ph.D,is a senior member of China Computer Fe-deration.His main research interests include spatio-temporal data mining,urban computing and federated learning.
  • Supported by:
    National Natural Science Foundation of China(72061127001),Natural Science Foundation of Beijing,China(4212021) and Beijing Nova Program(Z201100006820053).

PDF (PC)

6052

Abstract: Federated learning is proposed to solve the contradiction between data sharing and privacy-preserving.It aims to build collaborative models by securely interacting irreversible information (e.g.,model parameters or gradient updates).However,the risks of privacy leakage and malicious attacks in the process of model local training,information interaction and parameter transmission have brought major challenges to the practical application of federated learning.This paper summarizes the Attack beha-viors and corresponding defense strategies in the modeling and deployment process of federated learning.Firstly,this paper briefly reviews the development process of federated learning and the basic modeling process.Next,it classifies attack behaviors in fede-ral learning training and deployment from three aspects:confidentiality,availability and integrity,and combs the latest research on privacy theft and malicious attacks.Then,it summarizes defense countermeasures from two directions: honest-but-curious attac-kers and malicious attackers,and analyzes the defense capabilities of different strategies.Finally,it presents some discussions about the problems and challenges of attack and defense methods in the practice of federated learning.Besides,it looks forward to their future development direction of federated learning in defense strategy and system design.

Key words: Attacks and Defenses, Data fusion, Data security, Federated learning, Machine learning

CLC Number: 

  • TP391
[1]LI T,SAHU A K,TALWALKAR A,et al.Federated learning:Challenges,methods,and future directions[J].IEEE Signal Processing Magazine,2020,37(3):50-60.
[2]GOODMAN B,FLAXMAN S.European Union regulations onalgorithmic decision-making and a “rightto explanation”[J].AI Magazine,2017,38(3):50-57.
[3]GANJU K,WANG Q,YANG W,et al.Property inference attacks on fully connected neural networks using permutation invariant representations[C]//Proceedings of ACM SIGSAC.2018:619-633.
[4]ATENIESE G,MANCINIL V,SPOGNARDI A,et al.Hacking smart machines with smarter ones:How to extract meaningful data from machine learning classifiers[J].International Journal of Security and Networks,2015,10(3):137-150.
[5]MELIS L,SONG C,DE CRISTOFARO E,et al.Exploiting unintended feature leakage in collaborative learning[C]//IEEE Symposium on Security and Privacy.2019:691-706.
[6]SHOKRI R,STRONATI M,SONG C,et al.Membership inference attacks against machine learning models[C]//IEEE Symposium on Security and Privacy.2017:3-18.
[7]FREDRIKSON M,JHA S,RISTENPART T.Model inversion attacks that exploit confidence information and basic countermeasures[C]//Proceedings of ACM SIGSAC.2015:1322-1333.
[8]HITAJ B,ATENIESE G,PEREZ-CRUZ F.Deep models under the GAN:information leakage from collaborative deep learning[C]//Proceedings of ACM SIGSAC.2017:603-618.
[9]TRAMÉR F,ZHANG F,JUELS A,et al.Stealing machinelearning models via prediction apis[C]//USENIX Security.2016:601-618.
[10]PAPERNOT N,MCDANIEL P,SINHA A,et al.Sok:Secu-rityand privacy in machine learning[C]//IEEE European Symposium on Security and Privacy.2018:399-414.
[11]SZEGEDY C,ZAREMBA W,SUTSKEVER I,et al.Intriguing properties of neural networks[C]//ICLR,2014:1-11.
[12]GOODFELLOW I J,SHLENS J,SZEGEDY C.Explaining and harnessing adversarial examples[J].arXiv:1412.6572,2014.
[13]LI Y,WU B,JIANG Y,et al.Backdoor learning:A survey[J].arXiv:2007.08745,2020.
[14]WANG H,SREENIVASAN K,RAJPUT S,et al.Attack of the tails:Yes,you really can backdoor federated learning[J].arXiv:2007.05084,2020.
[15]SUN Z,KAIROUZ P,SURESH A T,et al.Can you really backdoor federated learning?[J].arXiv:1911.07963,2019.
[16]CHEN D,FU A,ZHOU C,et al.Federated Learning Backdoor Attack Scheme Based on Generative Adversarial Network[J].Journal of Computer Research and Development,2021,58(11):2364-2373.
[17]GARBER L.Denial-of-service attacks rip the Internet[J].Computer,2000,33(4):12-17.
[18]GU T,LIU K,DOLAN-GAVITT B,et al.Badnets:Evaluating backdooring attacks on deep neural networks[J].IEEE Access,2019,7:47230-47244.
[19]RAKIN A S,HE Z,FAN D.Tbt:Targeted neural network attack with bit trojan[C]//Proceedings of the IEEE Conference on CVPR.2020:13198-13207.
[20]XU R,BARACALDO N,ZHOU Y,et al.Hybridalpha:An efficient approach for privacy-preserving federated learning[C]//Workshop on Artificial Intelligence and Security.2019:13-23.
[21]AONO Y,HAYASHI T,WANG L,et al.Privacy-preservingdeep learning via additively homomorphic encryption[J].IEEE Transactions on Information Forensics and Security,2017,13(5):1333-1345.
[22]GEYER R C,KLEIN T,NABI M.Differentially private federated learning:A client level perspective[J].arXiv:1712.07557,2017.
[23]MCMAHAN H B,RAMAGE D,TALWAR K,et al.Learning differentially private recurrent language models[C]// ICLR.2017:171-182.
[24]TRUEX S,BARACALDO N,ANWAR A,et al.A hybrid approach to privacy-preserving federated learning[C]// Workshop on Artificial Intelligence and Security.2019:1-11.
[25]LIU Y,XIE Y,SRIVASTAVA A.Neural trojans[C]//International Conference on Computer Design.2017:45-48.
[26]TRAN B,LI J,MADRY A.Spectral signatures in backdoor attacks[J].arXiv:1811.00636,2018.
[27]WANG Z,SONG M,ZHANG Z,et al.Beyond inferring class representatives:User-level privacy leakage from federated lear-ning[C]//IEEE INFOCOM.2019:2512-2520.
[28]FREDRIKSON M,LANTZ E,JHA S,et al.Privacy in pharmacogenetics:An end-to-end case study of personalized warfarin dosing[C]//USENIX Security.2014:17-32.
[29]GEIPING J,BAUERMEISTER H,DRÖGE H,et al.Inverting Gradients--How easy isit to break privacy in federated learning?[C]//Advances in Neural Information Processing Systems.2020:16937-16947.
[30]ZHU L,HAN S.Deep leakage from gradients[C]//Advances in Neural Information Processing Systems.2019:17-31.
[31]PAPERNOT N,MCDANIEL P,GOODFELLOW I,et al.Practical black-box attacks against machine learning[C]//Procee-dings of ACM on Asia Conference on CCS.2017:506-519.
[32]JUUTI M,SZYLLER S,MARCHAL S,et al.PRADA:protecting against DNN model stealing attacks[C]//IEEE European Symposium on Security and Privacy.2019:512-527.
[33]OREKONDY T,SCHIELE B,FRITZ M.Knockoff nets:Stea-ling functionality of black-box models[C]//Proceedings of IEEE Conference on CVPR.2019:4954-4963.
[34]BHAGOJI A N,CHAKRABORTY S,MITTAL P,et al.Analyzing federated learning through an adversarial lens[C]//International Conference on Machine Learning.2019:634-643.
[35]TANG R,DU M,LIU N,et al.An embarrassingly simple approach for trojan attack in deep neural networks[C]//Procee-dings of ACM SIGKDD.2020:218-228.
[36]DUMFORD J,SCHEIRER W.Backdooring convolutional neural networks via targeted weight perturbations[C]//IEEE International Joint Conference on Biometrics.2020:1-9.
[37]BAGDASARYAN E,VEIT A,HUA Y,et al.How to backdoor federated learning[C]//International Conference on Artificial Intelligence and Statistics.2020:2938-2948.
[38]LIU Y,MA S,AAFER Y,et al.Trojaning attack on neural networks[C]//Proceedings Network and Distributed System Security Symposium.2018:1-11.
[39]CHEN X,LIU C,LI B,et al.Targeted backdoor attacks on deep learning systems using data poisoning[J].arXiv:1712.05526,2017.
[40]SU J,VARGAS D V,SAKURAI K.One pixel attack for fooling deep neural networks[J].IEEE Transactions on Evolutionary Computation,2019,23(5):828-841.
[41]FANG M,CAO X,JIA J,et al.Local modelpoisoning attacks to byzantine-robust federated learning[C]//USENIX Security.2020:1605-1622.
[42]XIE C,KOYEJO O,GUPTA I.Fall of empires:Breaking Byzantine-tolerant SGD by inner product manipulation[C]//Uncertainty in Artificial Intelligence.2020:261-270.
[43]BLANCHARD P,EL MHAMDI E M,GUERRAOUI R,et al.Machine learning with adversaries:Byzantine tolerant gradient descent[C]//Neural Information Processing Systems.2017:118-128.
[44]FUNG C,YOON C J M,BESCHASTNIKH I.The limitations of federated learning in sybil settings[C]//International Sympo-sium on Research in Attacks.2020:301-316.
[45]BHOWMICK A,DUCHI J,FREUDIGER J,et al.Protectionagainst reconstruction and its applications in private federated learning[J].arXiv:1812.00984,2018.
[46]SHOKRI R,SHMATIKOV V.Privacy-preserving deep learning[C]//Proceedings of ACM SIGSAC.2015:1310-1321.
[47]LIN Y,HAN S,MAO H,et al.Deep gradient compression:Reducing the communication bandwidth for distributed training[C]//ICLR.2017:1-12.
[48]AGRAWAL N,SHAHIN SHAMSABADI A,KUSNER M J,et al.QUOTIENT:two-party secure neural network training and prediction[C]//Proceedings of ACM SIGSAC.2019:1231-1247.
[49]ZHANG C,LI S,XIA J,et al.Batchcrypt:Efficient homomorphic encryption for cross-silo federated learning[C]//USENIX Security.2020:493-506.
[50]LIU C,CHAKRABORTY S,VERMA D.Secure model fusionfor distributed learning using partial homomorphic encryption[J].Policy-Based Autonomic Data Governance,2019(11550):154-179.
[51]HARDY S,HENECKA W,IVEY-LAW H,et al.Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption[J].arXiv:1711.10677,2017.
[52]HAO M,LI H,XU G,et al.Towards efficient and privacy-preserving federated deep learning[C]//IEEE International Confe-rence on Communications.2019:1-6.
[53]SAMANGOUEI P,KABKAB M,CHELLAPPA R.Defense-gan:Protecting classifiers against adversarial attacks using ge-nerative models[C]//ICLR.2018:185-195.
[54]UDESHI S,PENG S,WOO G,et al.Model agnostic defenceagainst backdoor attacks in machinelearning[J].arXiv:1908.02203,2019.
[55]GUO C,RANA M,CISSE M,et al.Countering adversarial images using input transformations[C]//ICLR.2017:1-13.
[56]DZIUGAITE G K,GHAHRAMANI Z,ROY D M.A study of the effect of jpg compression on adversarial images[J].arXiv:1608.00853,2016.
[57]HUANG R,XU B,SCHUURMANS D,et al.Learning with a strong adversary[C]//ICLR.2017:151-161.
[58]FUNG C,YOON C J M,BESCHASTNIKH I.Mitigating sybils in federated learning poisoning[J].arXiv:1808.04866,2018.
[59]SHEN S,TOPLE S,SAXENA P.Auror:Defending against poisoning attacks in collaborative deep learning systems[C]//Annual Conference on Computer Security Applications.2016:508-519.
[60]YIN D,CHEN Y,KANNAN R,et al.Byzantine-robust distributed learning:Towards optimal statistical rates[C]//InternationalConference on Machine Learning.2018:5650-5659.
[61]VILLARREAL-VASQUEZ M,BHARGAVA B.Confoc:Con-tent-focus protection against trojan attacks on neural networks[J].arXiv:2007.00711,2020.
[62]LIU K,DOLAN-GAVITT B,GARG S.Fine-pruning:Defendingagainst backdooring attacks on deep neural networks[C]//International Symposium on Research in Attacks,Intrusions,and Defenses.2018:273-294.
[63]WANG B,YAO Y,SHAN S,et al.Neural cleanse:Identifying and mitigating backdoor attacks in neural networks[C]//IEEE Symposium on Security and Privacy.2019:707-723.
[64]WANG J Z,KONG L W,HUANG Z C,et al.Research advances on privacy protection of federated learning[J].Big data research,2021,7(3):139-149.
[65]CHEN B,XIE Y Y,ZHANG J L,et al.Survey of security and privacy in federated learning[J].Journal of NUAA,2020,52(5):675-684.
[66]YANG Q.AI and Data Privacy Protection:The Way to Federated Learning[J].Journal of Information Security Research,2019,5(11):961-965.
[67]LI T,SAHU A K,ZAHEER M,et al.Federated optimization in heterogeneous networks[C]//Proceedings of Machine Learning and Systems.2020:429-450.
[68]MCMAHAN B,MOORE E,RAMAGE D,et al.Communica-tion-efficient learning of deep networks from decentralized data[C]//Artificial intelligence and statistics.2017:1273-1282.
[69]DWORK C,NAOR M.On the difficulties of disclosure prevention in statistical databases or the case for differential privacy[J].Journal of Privacy and Confidentiality,2010,2(1):1-12.
[70]DWORK C.Differential privacy:A survey of results[C]//International conference on theory and applications of models of computation.2008:1-19.
[71]BONAWITZ K,IVANOV V,KREUTER B,et al.Practical secure aggregation for privacy-preserving machine learning[C]//Proceedings of ACM SIGSAC.2017:1175-1191.
[72]YAO A C.Protocols for secure computations[C]//Annual Symposium on Foundations of Computer Science.1982:160-164.
[73]LIU D C,NOCEDAL J.On the limited memory BFGS method for large scale optimization[J].Mathematical Programming,1989,45(1):503-528.
[74]HE K,ZHANG X,REN S,et al.Deep residual learning forimage recognition[C]//Proceedings of IEEE Conference on CVPR.2016:770-778.
[75]KINGMA D P,BA J.Adam:A Method for Stochastic Optimization[C]//ICLR.2015:1-13.
[76]PAPERNOT N,MCDANIEL P,WU X,et al.Distillation as a defense to adversarial perturbations against deep neural networks[C]//IEEE Symposium on Security and Privacy.2016:582-597.
[77]BONAWITZ K,EICHNER H,GRIESKAMP W,et al.Towards federated learning at scale:System design[C]//IEEE CVPR Workshop.2019:3310-3319.
[78]DWORK C,ROTH A.The algorithmic foundations of differential privacy[J].Foundations and Trends in Theoretical Compu-ter Science,2014,9(3/4):211-407.
[79]FRIEDMAN A,SCHUSTER A.Data mining with differential privacy[C]//Proceedings of ACM SIGKDD.2010:493-502.
[80]ZHANG J,ZHANG Z,XIAO X,et al.Functional mechanism:regression analysis under differential privacy[J].arXiv:1208.0219,2012.
[81]ABADI M,CHU A,GOODFELLOW I,et al.Deep learning with differential privacy[C]//Proceedings of ACM SIGSAC.2016:308-318.
[82]LI R J,JIA C F,WANG Y F.Multi-key homomorphic proxy re-encryption scheme based on NTRU and its application[J].Journal on Communications,2021,42(3):11-22.
[83]ARMKNECHT F,BOYD C,CARR C,et al.A Guide to Fully Homomorphic Encryption[J].IACR Cryptology EPrint Archive,2015(1192):1-35.
[84]PAILLIER P.Public-key cryptosystems based on composite degree residuosity classes[C]//International Conference on the Theory and Applications of Cryptographic Techniques.1999:223-238.
[85]LINDNER R,PEIKERT C.Better key sizes (and attacks) for LWE-based encryption[C]//Cryptographers’ Track at the RSA Conference.2011:319-339.
[86]DAMGÅRD I,JURIK M.A generalisation,a simplification and some applications of Paillier’s probabilistic public-key system[C]//International Workshop on Public Key Cryptography.2001:119-136.
[87]DOAN B G,ABBASNEJAD E,RANASINGHE D C.Februus:Input purification defense against trojan attacks on deep neural network systems[C]//Annual Computer Security Applications Conference.2020:897-912.
[88]SELVARAJU R R,COGSWELL M,DAS A,et al.Grad-cam:Visual explanations from deep networks via gradient-based localization[C]//Proceedings of IEEE Conference on ICCV.2017:618-626.
[89]HUANG X,ALZANTOT M,SRIVASTAVA M.Neuronin-spect:Detecting backdoors in neural networks via output explanations[J].arXiv:1911.07399,2019.
[90]CHOQUETTE-CHOO C A,DULLERUD N,DZIEDZIC A,et al.CaPC Learning:Confidential and Private Collaborative Learning[C]//ICLR.2021:212-223.
[91]ADI Y,BAUM C,CISSE M,et al.Turning your weakness into astrength:Watermarking deep neural networks by backdooring [C]//USENIX Security.2018:1615-1631.
[1] LU Chen-yang, DENG Su, MA Wu-bin, WU Ya-hui, ZHOU Hao-hao. Federated Learning Based on Stratified Sampling Optimization for Heterogeneous Clients [J]. Computer Science, 2022, 49(9): 183-193.
[2] LENG Dian-dian, DU Peng, CHEN Jian-ting, XIANG Yang. Automated Container Terminal Oriented Travel Time Estimation of AGV [J]. Computer Science, 2022, 49(9): 208-214.
[3] NING Han-yang, MA Miao, YANG Bo, LIU Shi-chang. Research Progress and Analysis on Intelligent Cryptology [J]. Computer Science, 2022, 49(9): 288-296.
[4] TANG Ling-tao, WANG Di, ZHANG Lu-fei, LIU Sheng-yun. Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy [J]. Computer Science, 2022, 49(9): 297-305.
[5] HE Qiang, YIN Zhen-yu, HUANG Min, WANG Xing-wei, WANG Yuan-tian, CUI Shuo, ZHAO Yong. Survey of Influence Analysis of Evolutionary Network Based on Big Data [J]. Computer Science, 2022, 49(8): 1-11.
[6] LI Yao, LI Tao, LI Qi-fan, LIANG Jia-rui, Ibegbu Nnamdi JULIAN, CHEN Jun-jie, GUO Hao. Construction and Multi-feature Fusion Classification Research Based on Multi-scale Sparse Brain Functional Hyper-network [J]. Computer Science, 2022, 49(8): 257-266.
[7] ZHANG Guang-hua, GAO Tian-jiao, CHEN Zhen-guo, YU Nai-wen. Study on Malware Classification Based on N-Gram Static Analysis Technology [J]. Computer Science, 2022, 49(8): 336-343.
[8] XIAO Zhi-hong, HAN Ye-tong, ZOU Yong-pan. Study on Activity Recognition Based on Multi-source Data and Logical Reasoning [J]. Computer Science, 2022, 49(6A): 397-406.
[9] YAN Meng, LIN Ying, NIE Zhi-shen, CAO Yi-fan, PI Huan, ZHANG Lan. Training Method to Improve Robustness of Federated Learning [J]. Computer Science, 2022, 49(6A): 496-501.
[10] YAO Ye, ZHU Yi-an, QIAN Liang, JIA Yao, ZHANG Li-xiang, LIU Rui-liang. Android Malware Detection Method Based on Heterogeneous Model Fusion [J]. Computer Science, 2022, 49(6A): 508-515.
[11] LI Ya-ru, ZHANG Yu-lai, WANG Jia-chen. Survey on Bayesian Optimization Methods for Hyper-parameter Tuning [J]. Computer Science, 2022, 49(6A): 86-92.
[12] ZHAO Lu, YUAN Li-ming, HAO Kun. Review of Multi-instance Learning Algorithms [J]. Computer Science, 2022, 49(6A): 93-99.
[13] LU Chen-yang, DENG Su, MA Wu-bin, WU Ya-hui, ZHOU Hao-hao. Clustered Federated Learning Methods Based on DBSCAN Clustering [J]. Computer Science, 2022, 49(6A): 232-237.
[14] WANG Fei, HUANG Tao, YANG Ye. Study on Machine Learning Algorithms for Life Prediction of IGBT Devices Based on Stacking Multi-model Fusion [J]. Computer Science, 2022, 49(6A): 784-789.
[15] XU Jie, ZHU Yu-kun, XING Chun-xiao. Application of Machine Learning in Financial Asset Pricing:A Review [J]. Computer Science, 2022, 49(6): 276-286.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
渝ICP备16013121号-4
Add:18# Honghu West Rd., Yubei ,Chongqing,China    Post Code: 401121
Tel: 023-63500828/023-67039612/023-67039623
E-mail: jsjkx12@163.com
Powered by Beijing Magtech Co., Ltd.