Abstract: Due to high data pre-processing costs and missing local features detection in self-distillation methods for models compression,a similarity and consistency by self-distillation(SCD) method is proposed to improve model classification accuracy.Firstly,different layers of the sample images are learned to get the feature maps,and the attention maps are obtained by the distribution of feature weights.Then,the similarity of the attention graph between samples within the mini-batch is calculated to obtain the similar consistency knowledge matrix,and the similar consistency-based knowledge is constructed without distorting the instance data or extracting the same class of data to obtain additional inter-instance knowledge,avoiding a large amount of data pre-processing work.Finally,the similar consistency knowledge matrix is passed unidirectionally between intermediate layers of the model,allowing shallow layers to mimic deep layers and capture richer contextual scenes and local features which can solve the problem of missing local feature detection.Experimental results show that the proposed SCD method can improve the classification accuracy on the public dataset CIFAR100.Compared with the self attention distillation(SAD) method and the similarity-preserving knowledge distillation(SPKD) method,the average improvement is 1.42%.Compared with the be your own teacher(BYOT) method and the on-the-fly native ensemble(ONE) method,the average improvement is 1.13%.Compared with the data-distortion guided self-distillation(DDGSD) method and the class-wise self-knowledge distillation(CS-KD) method,the average improvement is 1.23%.

Key words: Knowledge distillation, Knowledge representation, Self-distillation, Similarity consistency, Knowledge matrix