关键词: 大语言模型, 模型剪枝, Taylor+重要性估计, LoRa

Abstract: Aiming at the problems of high PPL( perplexity ),low text generation accuracy and slow model reasoning speed in Zero-shot Performance after the existing large-scale language model is processed by pruning algorithm,this paper proposes a pruning metric algorithm LAM based on the joint magnitude of loss.In the process of estimating the weight importance,the loss function information and the weight activation information are fused.By using the LAM algorithm,the limitations caused by the omission of the second derivative in the Taylor expansion of the gradient information in the process of weight importance evaluation are eliminated,and the accuracy and robustness of the model pruning process are improved.Enhance the versatility of the pruning algorithm.When establishing the coupling structure,a single coupling structure is proposed,and the neurons in the multi-layer perceptron( MLP ) in the Transformer block are selected as the initial trigger.Only the attention layer,the query vector,the key vector,and the value vector layer are considered to activate the neurons to establish the coupling structure.Thus,the number of parameters required to identify the coupling structure group is reduced,and the pruning speed and throughput are improved.The Zero-shot Performance experiments on WikiText2 dataset and PTB dataset show that when the pruning rate is 25 %,the PPL scores of LLaMA-7B are 20.24 and 36.05,respectively,which are lower than other pruning algorithms.The PPL scores of Vicuna-7B after pruning are 21.24 and 85.81,which are also better than other pruning algorithms,showing that the algorithm has higher universality and accuracy.

Key words: Large language models, Model pruning, Taylor+ importance estimates, LoRa