Abstract: As summarizations of the experiences of practical software design,design patterns are regarded as an effective means for software design assistance.Most of the current researches on design patterns mining aim at recognition of design pattern instance in source codes,modelling design patterns with natural language corpus is largely unexplored.In order to enhance the performance of language model for recommending design patterns with codes,class diagram or object collaboration,a design pattern classification mining model based on CodeBERT,named dpCodeBERT,is proposed,achieving the contrast understanding of design patterns in natural language and programming language.Firstly,multi-classification dataset and code search dataset are ge-nerated using random combination and used as inputs of the model.Using dpCodeBERT to get attention weights of each layer of transformer of each token and statement from the inputs.Secondly,the input dataset is further improved by analyzing attention weights and discovering the most important category of inputs.Finally,dpCodeBERT is applied to specific software engineering downstream tasks such as design patterns selection and design patterns code search.The purposes of tasks are accomplished by mapping distributed features to sample space trough fully connected layers and outputting multi values.The result of the experiment on 80 software design problems in design pattern selection task shows that ratio of correct detection of design pattern(RCDDP)and mean reciprocal rank(MRR) of dpCodeBERT are improved by the average of 10%~20% compared with baseline mo-dels,and the design pattern selection is more accurate.Through in-depth study of the data demand of the model,dpCodeBERT improves the understanding of class code of CodeBERT and discovers the application of CodeBERT in design patterns mining.It has the characteristics of accurate prediction and great scalability.

Key words: Design pattern mining, Natural language processing, Pre-trained language models, CodeBERT, Model fine-tuning, Vector quantization