关键词: 联邦学习, 差分隐私, 知识蒸馏, 隐私保护, 隐私预算

Abstract: Differential privacy technology,as a privacy protection method,has been widely applied in federated learning.The existing research on the application of differential privacy in federated learning either fails to consider unlabeled public data or the difference in data volume between clients,which limits its application in real-world scenarios.This paper proposes a differential privacy federated learning method based on knowledge distillation,which introduces unlabeled public datasets and considers the differences in data volume between clients.A dedicated differential privacy scheme is designed for this scenario.Firstly,the clients are grouped into “large data clients” and “general clients” based on the size of the data.The teacher model is trained using the data from the large data clients,and the teacher model adds pseudo labels to the public dataset.Then,the public dataset is used as a “special client” to jointly conduct federated training with the “general client”.Adopting differential privacy technology to ensure the data privacy of clients,as the data of special clients only involves privacy with labels,more privacy budgets are allocated to them in federated training compared to general clients.Limit the total amount of privacy budget,set the privacy budget for the federal training stage as a fixed value,and adjust the privacy budget for the pseudo label addition stage based on the client’s privacy needs and the parallel combination property of privacy budget.Experiments on the MNIST and SVHN datasets show that,under the same privacy budget consumption,the trained model has higher accuracy than traditional methods.This scheme has scalability,and its high flexibility of privacy budget allocation enables it to meet complex privacy needs.

Key words: Federated learning, Differential privacy, Knowledge distillation, Privacy protection, Privacy budget