关键词: 噪声标签, 置信学习, 主动学习, 知识蒸馏, 自蒸馏

Abstract: Text classification is an important and classic problem in the field of natural language processing,and it is often used in news classification,sentiment analysis and other scenarios.The existing deep learning-based classification methods have the following three problems:1)There are a large number of noisy labels in real-life datasets,and directly using these data to train the model will seriously affect the performance of the model.2)With the introduction of the pre-training model,the accuracy of model classification has improved,but the scale of the model and the number of inference calculations have also increased significantly,which make it a challenge to use pre-training models on devices with limited resources.3)The pre-training model has a large number of redundant calculations,which will lead to low prediction efficiency when the amount of data is large.To address these issues,this paper proposes a text classification method that combines anti-noise and double distillation(including knowledge distillation and self-distillation).Through the threshold anti-noise method based on confidence learning and a new active learning sample selection algorithm,the quality of the data is improved with a small amount of labeling cost.Meanwhile,the combination of knowledge distillation and self-fistillation reduces the scale of the model and redundant calculation,thereby it can flexibly adjust the inference speed according to the demand.Extensive experiments are performed on real datasets to evaluate the performance of the proposed method.Experimental results show that the accuracy of the proposed method after anti-noise increases by 1.18%,and it can be 4～8 times faster than BERT under small accuracy losses.

Key words: Noise label, Confidence learning, Active learning, Knowledge distillation, Self-distillation