基于语义驱动执行的代码质量评测框架

doi:10.11896/jsjkx.250900140

计算机科学 ›› 2026, Vol. 53 ›› Issue (10): 320-335.doi: 10.11896/jsjkx.250900140

基于语义驱动执行的代码质量评测框架

张赛¹, 许方舟¹, 张小旺¹, 韩亚洪¹, 邢振昌², 冯志勇¹

1 天津大学智能与计算学部天津 300354
2 联邦科学与工业研究组织Data61 澳大利亚堪培拉 2603

收稿日期:2025-09-24 修回日期:2026-01-24 发布日期:2026-10-13
通讯作者: 张小旺(xiaowangzhang@tju.edu.cn)
作者简介:(zhang_sai@tju.edu.cn)
基金资助:
黔科合重大专项字([2024]002)

Semantic-driven Execution-based Framework for Code Quality Evaluation

ZHANG Sai¹, XU Fangzhou¹, ZHANG Xiaowang¹, HAN Yahong¹, XING Zhenchang², FENG Zhiyong¹

1 College of Intelligence and Computing,Tianjin University,Tianjin 300354, China
2 Data61,Commonwealth Scientific and Industrial Research Organisation,Canberra 2603,Australian

Received:2025-09-24 Revised:2026-01-24 Online:2026-10-13
About author:ZHANG Sai,born in 1994,postgra-duate.Her main research interests include intelligent software engineering,large language models,and software testing.
ZHANG Xiaowang,born in 1980,Ph.D,professor,Ph.D supervisor,is a senior member of CCF(No.48571S).His main research interests include knowledge graphs,natural language processing,graph database systems,and software security.
Supported by:
Guizhou Provincial Major Scientific and Technological Program([2024]002).

摘要/Abstract

摘要： 代码质量评测是在特定任务下对代码的质量进行自主度量与评分的过程。现有评测技术大致分为基于匹配和基于执行两种方式。前者侧重在代码文本层面与参照代码进行静态比对,难以全面评测代码行为语义上的差异。后者依赖大量手工或半自动构造的测试用例执行评测,构建成本高且覆盖范围受限。随着大语言模型(Large Language Model,LLM)的快速发展,越来越多工作尝试将 LLM 作为自动评测器,然而在缺乏完备测试用例和高质量参照代码的现实环境中,如何减轻LLM推理中的幻觉、获得与专家判定及真实执行表现高度一致的自动评测结果,仍然是亟待解决的关键问题。为此,提出基于语义驱动执行的代码质量评测框架SDEval及其细粒度评分变体SDEval-FG。该框架首先通过代码语义递归理解,将复杂程序拆分为细粒度语义单元并逐层推理,以强化对程序中控制流与数据流关系的理解能力,捕获代码片段之间的逻辑依赖,抑制幻觉产生。随后通过自动测试用例生成与补全模块,构建动态语义驱动执行视角,使LLM兼顾语义正确性与功能完整性,从而弥合代码文本与实际执行效果之间的鸿沟。相对于仅针对提升测试用例生成与覆盖的现有方法来说,SDEval将评测重心从生成高质量测试用例,转为面向缺陷类别的可解释评分,以提高与人类专家判定的相关性。实验结果表明,在HumanEval数据集上,SDEval相较于基于执行的ChatTester在Pearson相关系数上提升了11.86%。在CodePair数据集上,变体SDEval-FG相较于CoT提示基线在Pearson指标上提升了31.4%。

关键词: 代码质量评测, 代码语义表示, 测试用例生成, 缺陷识别, 大语言模型

Abstract: Code quality evaluation is the process of autonomously measuring and scoring code quality for specific tasks.Existing evaluation techniques can be broadly divided into match-based and execution-based approaches.The former emphasizes static textual comparison with reference code,making it difficult to assess semantic differences in program behavior comprehensively.The latter depends on large numbers of manually or semi-automatically constructed test cases,which are costly to build and have limi-ted coverage.With the rapid development of large language models(LLMs),an increasing number of studies are using LLMs as automatic evaluators.However,in realistic scenarios where complete test suites and high-quality reference code are often unavailable,how to mitigate hallucinations in LLM reasoning and obtain automated evaluation results that are highly consistent with expert judgments and actual execution behavior remains a key open problem.To this end,this paper proposes SDEval,a semantic-driven execution framework for code quality evaluation,together with its fine-grained scoring variant SDEval-FG.Firstly,the framework performs recursive semantic understanding,decomposing complex programs into fine-grained semantic units and conducting layered reasoning to strengthen comprehension of control-flow and data-flow relationships,capture logical dependencies among code fragments,and suppress hallucinations.It then constructs a dynamic,semantic-driven execution perspective through automatic test-case generation and completion,enabling the LLM to consider both semantic correctness and functional completeness,thereby bridging the gap between code text and actual execution behavior.Compared with existing methods that mainly aim to improve test-case generation and coverage,SDEval shifts the evaluation focus from generating high-quality test cases to interpretable,defect-category-oriented scoring,thus enhancing its correlation with human expert judgments.Experimental results show that on the HumanEval dataset,SDEval improves the Pearson correlation coefficient by 11.86% over the execution-based ChatTester,and on the CodePair dataset,the SDEval-FG variant achieves a 31.4% improvement in Pearson correlation over the CoT-prompting baseline.

Key words: Code quality evaluation, Code semantic representation, Test case generation, Defect identification, Large language models

中图分类号:

TP311

张赛, 许方舟, 张小旺, 韩亚洪, 邢振昌, 冯志勇. 基于语义驱动执行的代码质量评测框架[J]. 计算机科学, 2026, 53(10): 320-335. https://doi.org/10.11896/jsjkx.250900140

ZHANG Sai, XU Fangzhou, ZHANG Xiaowang, HAN Yahong, XING Zhenchang, FENG Zhiyong. Semantic-driven Execution-based Framework for Code Quality Evaluation[J]. Computer Science, 2026, 53(10): 320-335. https://doi.org/10.11896/jsjkx.250900140

参考文献

[1] ZHUO T Y.ICE-Score:Instructing Large Language Models to Evaluate Code [C]//EACL 2024.ACL,2024:2232-2242.
[2] REN S,GUO D,LU S,et al.CodeBLEU:A Method for Automatic Evaluation of Code Synthesis [PP/OL].arXiv(2020-09-22)[2026-08-19].https://arxiv.org/abs/2009.10297.
[3] LIU K,HAN Y,ZHANG J M,et al.Who Judges the Judge:AnEmpirical Study on Online Judge Tests [C]//Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis(ISSTA 2023).ACM,2023:334-346.
[4] LU S,GUO D,REN S,et al.CodeXGLUE:A Machine Learning Benchmark Dataset for Code Understanding and Generation [C]//NeurIPS 2021 Datasets and Benchmarks.Curran Asso-ciates,2021.
[5] LE H,WANG Y,GOTMARE A D,et al.CodeRL:Mastering Code Generation Through Pretrained Models and Deep Reinforcement Learning [J].Advances in Neural Information Processing Systems,2022,35:21314-21328.
[6] ZAN D,CHEN B,ZHANG F,et al.Large Language ModelsMeet NL2Code:A Survey [C]//ACL 2023.ACL,2023:7443-7464.
[7] WANG D Y,MO R,ZHAN W J,et al.Analysis of the Code Quality of the Code Automatic Generation Tool GitHub Copilot [J].Computer Science,2025,52(7):37-49.
[8] KULAL S,PASUPAT P,CHANDRA K,et al.SPoC:Search-Based Pseudocode to Code [C]//Advances in Neural Information Processing Systems 32(NeurIPS 2019).Curran Associates,2019.
[9] HENDRYCKS D,BASART S,KADAVATH S,et al.Measu-ring Coding Challenge Competence with APPS [PP/OL]. arXiv(2021-05-20)[2026-08-19].https://doi.org/10.48550/arXiv.2105.09938.
[10] ZHOU S,ALON U,AGARWAL S,et al.CodeBERTScore:Evaluating Code Generation with Pretrained Models of Code [C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing(EMNLP 2023).ACL,2023.
[11] LI K,YUAN Y.Large Language Models as Test Case Generators:Performance Evaluation and Enhancement [PP/OL].arXiv(2024-04-20)[2026-08-19].https://arxiv.org/abs/2404.13340.
[12] YUAN Z,LOU Y,LIU M,et al.No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation [PP/OL].arXiv(2023-05-07)[2026-08-19].https://arxiv.org/abs/2305.04207.
[13] PAPINENI K,ROUKOS S,WARD T,et al.BLEU:a Method for Automatic Evaluation of Machine Translation [C]//Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics.ACL,2002:311-318.
[14] LIN C Y.ROUGE:A Package for Automatic Evaluation ofSummaries [C]//Text Summarization Branches Out.ACL,2004:74-81.
[15] EVTIKHIEV M,BOGOMOLOV E,SOKOLOV Y,et al.Out of the BLEU:How Should We Assess Quality of the Code Generation Models? [J].Journal of Systems and Software,2023,203:111741.
[16] DEVLIN J,CHANG M W,LEE K,et al.BERT:Pre-training of Deep Bidirectional Transformers for Language Understanding [C]//Proceedings of NAACL-HLT 2019.ACL,2019:4171-4186.
[17] ACHIAM J,ADLER S,AGARWAL S,et al.GPT-4 Technical Report [PP/OL].arXiv(2023-12-19)[2026-08-19].https://arxiv.org/html/2303.08774v4.
[18] CHEN X T,YE J J,ZU C,et al.Robustness of GPT Family of Large Language Models in Natural Language Processing Tasks [J].Journal of Computer Research and Development,2024,61(5):1128-1142.
[19] JI Z,LEE N,FRIESKE R,et al.Survey of Hallucination in Na-tural Language Generation [J].ACM Computing Surveys,2023,55(12):1-38.
[20] CRESWELL A,SHANAHAN M.Faithful Reasoning UsingLarge Language Models [PP/OL].arXiv(2022-08-30)[2026-08-19].https://arxiv.org/abs/2208.14271.
[21] ZHONG M,LIU Y,YIN D,et al.Towards a Unified Multi-Dimensional Evaluator for Text Generation [C]//Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.ACL,2022:2023-2038.
[22] WANG Z P,HE T K,ZHAO R Y,et al.Exploration and Im-provement of Capabilities of LLMs in Code Refinement Task [J].Journal of Software,2025,36(6):2477-2500.
[23] KUKICH K.Techniques for Automatically Correcting Words inText [J].ACM Computing Surveys,1992,24(4):377-439.
[24] JACCARD P.Étude comparative de la distribution florale dans une portion des Alpes et du Jura [J].Bulletin de la Société Vaudoise des Sciences Naturelles,1901,37:547-579.
[25] DICE L R.Measures of the Amount of Ecologic Association Between Species [J].Ecology,1945,26(3):297-302.
[26] BANERJEE S,LAVIE A.METEOR:An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments [C]//ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization.ACL,2005:65-72.
[27] POPOVIČ M.chrF:Character N-gram F-score for AutomaticMT Evaluation [C]//Proceedings of the Tenth Workshop on Statistical Machine Translation.ACL,2015:392-395.
[28] CHEN M,TWOREK J,JUN H,et al.Evaluating large language models trained on code [PP/OL].arXiv[2026-09-11].https://arxiv.org/abs/2107.03374.
[29] LING W,BLUNSOM P,GREFENSTETTE E,et al.LatentPredictor Networks for Code Generation [C]//Proceedings of the 54th Annual Meeting of the ACL(ACL 2016).ACL,2016:599-609.
[30] CHEN M,TWOREK J,JUN H,et al.Evaluating Large Lan-guage Models Trained on Code [PP/OL].arXiv(2021-07-07)[2026-08-19].https://arxiv.org/abs/2107.03374.
[31] CADAR C,DUNBAR D,ENGLER D R,et al.KLEE:Unassis-ted and Automatic Generation of High-Coverage Tests for Complex Systems Programs [C]//Proceedings of the 8th USENIX Symposium on Operating Systems Design and Implementation(OSDI 2008).USENIX Association,2008:209-224.
[32] FRASER G,ARCURI A.EvoSuite:Automatic Test Suite Generation for Object-Oriented Software [C]//Proceedings of the 2011 European Software Engineering Conference and Symposium on the Foundations of Software Engineering(ESEC/FSE 2011).ACM,2011:416-419.
[33] XIA C S,PALTENGHI M,TIAN J L,et al.Fuzz4All:UniversalFuzzing with Large Language Models [C]//Proceedings of the 46th IEEE/ACM International Conference on Software Engineering(ICSE 2024).IEEE/ACM,2024.
[34] MENG R,XIA Z,MIAO H,et al.Large Language ModelGuided Protocol Fuzzing [C]//Network and Distributed System Security Symposium(NDSS 2024).Internet Society,2024.
[35] ZHAO X L,PAN X L,ZOU Y Z,et al.Research on Comment Quality Evaluation for Code Comment Generation Tasks [J].Journal of Software,2025,36(8):3744-3768.
[36] TRAN N,TRAN H,NGUYEN S,et al.Does BLEU ScoreWork for Code Migration? [C]//Proceedings of the 2019 IEEE/ACM 27th International Conference on Program Comprehension(ICPC).IEEE,2019:165-176.
[37] HU T,XU Z,FANG Y,et al.Fine-Grained Code Clone Detection with Block-Based Splitting of Abstract Syntax Tree [C]//Proceedings of the 32nd ACM SIGSOFT International Sympo-sium on Software Testing and Analysis(ISSTA 2023).ACM,2023:89-100.
[38] SHI E,WANG Y,DU L,et al.CoCoAST:Representing Source Code via Hierarchical Splitting and Reconstruction of Abstract Syntax Trees [J].Empirical Software Engineering,2023,28(6):1-41.
[39] CHOI Y,KIM H,LEE J H.BLOCSUM:Block Scope-BasedSource Code Summarization via Shared Block Representation [C]//Findings of the Association for Computational Linguistics:ACL 2023.ACL,2023:11427-11441.
[40] YU D Q,PENG X,ZHAO W Y.Automatic Refactoring Method of Clone Code Using Abstract Syntax Tree and Static Analysis [J].Small & Microcomputer Systems,2009,30(9):1752-1760.
[41] CASSANO F,GOUWAR J,NGUYEN D,et al.MultiPL-E:A Scalable and Polyglot Approach to Benchmarking Neural Code Generation [J].IEEE Transactions on Software Engineering,2023,49(7):3675-3691.
[42] PURI R,KUNG D S,JANSSEN G,et al.CodeNet:A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks [PP/OL].arXiv(2021-05-25)[2026-08-19].https://arxiv.org/abs/2105.12655.
[43] NAIR P R.Increasing Employability of Indian EngineeringGraduates Through Experiential Learning Programs and Competitive Programming:A Case Study [J].Procedia Computer Science,2020,172:831-837.
[44] DAS D,MATHEWS N S,CHIMALAKONDA S.Exploring Security Vulnerabilities in Competitive Programming:An Empirical Study [C]//Proceedings of the 26th International Confe-rence on Evaluation and Assessment in Software Engineering(EASE 2022).ACM,2022:110-119.
[45] WAN T,JUN H,PAN W,et al.Kappa Coefficient:A Popular Measure of Rater Agreement [J].Shanghai Archives of Psy-chiatry,2015,27(1):62-67.
[46] GILL G K,KEMERER C F.Cyclomatic Complexity Density and Software Maintenance Productivity [J].IEEE Transactions on Software Engineering,1991,17(12):1284-1288.

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

基于语义驱动执行的代码质量评测框架

Semantic-driven Execution-based Framework for Code Quality Evaluation

PDF (PC)

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 0

Metrics

本文评价

推荐阅读 0