Abstract: In recent years,machine learning techniques have made significant advancements in defect prediction within software development,enabling the automatic detection of errors in large-scale codebases.These advancements are expected to enhance the reliability,security,and overall quality of software.Defect prediction models can autonomously identify whether code contains errors.However,existing models,while having certain advantages,also exhibit limitations.They often fail to accurately identify vulnerabilities or incorrectly label defective code segments as problem-free.Currently,there is a lack of systematic empirical studies on the quality of defect detection models.The existing method,DPTester,assesses the effectiveness of defect models by generating defective code through modifications to if conditions in the code.However,the defect code produced by this method is overly simplistic,and the evaluation scenarios do not cover a wide range of models,including the latest large language models.To address this gap,this paper proposes an improved method called DefectGen.This new approach introduces multiple strategies to generate defect code that more closely reflects real-world issues.Furthermore,the evaluation of defect models includes large language mo-dels.Experimental results indicate that DefectGen significantly enhances the ability to generate complex defect code compared to previous methods,producing 1.2 times more defective code from a single correct code instance.When testing the CodeT5+,CodeBERT,and GPT-4o models,the proportions of incorrect defect predictions were found to be 62%,78%,and 30%.Additionally,DefectGen demonstrates higher efficiency in both test input generation and defect detection phases,with generation and detection times of 0.003 seconds and 0.02 seconds per test input.These results suggest that DefectGen not only effectively exposes the limitations of existing models but also provides new opportunities for improving defect prediction models and enhancing software quality assurance processes.

Key words: Defect prediction, Machine learning, Large language models