计算机科学 ›› 2026, Vol. 53 ›› Issue (10): 345-355.doi: 10.11896/jsjkx.250900141

• 计算机软件 • 上一篇    下一篇

面向安全关键软件单元测试的数据集智能生成方法

潘天乐, 薛雯耀, 任庆玮, 王轶辰   

  1. 北京航空航天大学可靠性与系统工程学院 北京 100191
  • 收稿日期:2025-09-24 修回日期:2025-12-19 发布日期:2026-10-13
  • 通讯作者: 王轶辰(wangyichen@buaa.edu.cn)
  • 作者简介:(pantianle@buaa.edu.cn)

Intelligent Dataset Generation Method for Safety-critical Software Unit Testing

PAN Tianle, XUE Wenyao, REN Qingwei, WANG Yichen   

  1. School of Reliability and Systems Engineering,Beihang University,Beijing 100191, China
  • Received:2025-09-24 Revised:2025-12-19 Online:2026-10-13
  • About author:PAN Tianle,born in 2003,postgra-duate.His main research interests include software intelligent testing technology,and so on.
    WANG Yichen,born in 1977,associate professor.His main research interests include model-based software testing,software quality evaluation,and so on.

摘要: 如果航空、航天等领域的安全关键软件质量得不到保障,会造成极大的危害性事故。软件测试,是保障软件质量的主要方式。单元测试作为软件测试的基础环节,是保证安全关键软件质量的必要步骤。单元测试的核心环节是测试用例设计,传统单元测试用例设计方法有人工编写和自动生成两类。大语言模型(LLM)等人工智能技术的发展,为单元测试用例设计提供了全新的智能生成思路。对于单元测试用例设计这类复杂任务,通用大语言模型通常不具备较高的适配度,使用数据集训练大语言模型是一种能有效改善模型任务表现且被广泛使用的方式。目前,面向模型训练的单元测试数据集大多使用静态分析等传统方式构建,存在映射关系有误、缺乏统一的定义与评估标准以及全面性和逻辑性不足的问题。针对上述问题,提出了结构化的单元测试数据集的全面定义和基于LLM的单元测试数据集的构建方法UTDCM-LLM。方法包括基于人工的单元测试数据点的构建、基于思维链的大语言模型提示词设计、基于LLM的单元测试数据集智能批量生成和基于静态检查的数据集评估四大环节。此外,还设计了基于数据集的模型微调实验。实验成功验证了单元测试数据集在模型训练上的有效性和UTDCM-LLM方法的可行性。

关键词: 安全关键软件, 单元测试, 大语言模型, 数据集, 模型训练

Abstract: For safety-critical software in domains such as aviation and aerospace,inadequate software quality can lead to highly hazardous accidents.Software testing is the primary means of ensuring software quality,and unit testing-being the foundational phase of software testing-is a necessary step for guaranteeing the quality of safety-critical software.The core component of unit testing is test case design.Traditional approaches to unit test case design fall into two categories:manual writing and automatic generation.Recent advances in artificial intelligence,particularly large language model(LLM),have introduced new intelligent strategies for generating unit test cases.However,general-purpose LLMs typically lack sufficient adaptability for complex tasks like unit test case design.A widely adopted and effective approach to improving model performance is training LLMs on specia-lized datasets.Currently,most unit test datasets used for model training are constructed using traditional techniques such as static analysis.These datasets often suffer from incorrect code-test mappings,lack standardized definitions and evaluation criteria,and are insufficient in both comprehensiveness and logical coherence.To address these issues,a structured,comprehensive definition for unit test datasets and a novel unit test dataset construction method by LLM named UTDCM-LLM are proposed.This method consists of four main components:manual construction of unit test data points,design of chain-of-thought prompts for LLMs,intelligent batch generation of unit test dataset using LLMs,and dataset evaluation based on static analysis.In addition,a model fine-tuning experiment based on the constructed dataset are conducted.The experiment successfully validates both the effectiveness of the dataset in model training and the feasibility of the UTDCM-LLM approach.

Key words: Safety-critical software, Unit testing, Large language model, Dataset, Model training

中图分类号: 

  • TP391
[1] LEVESON N G.Engineering a safer world:Systems thinkingapplied to safety[M].Cambridge:MIT Press,2016.
[2] ZHANG X H,HE Y Q.Overview of software testing methods[J].Science & Technology Vision,2012(4):35-37.
[3] MOHD-SHAFIE M L,NASIR WAN KADIR W M,KHATIB-SYARBINIM,et al.An EFSM-Based Test Data Generation Approach in Model-Based Testing[J].Computers,Materials & Continua,2022,71(3):4337-4354.
[4] SHANJ H,JIANG Y,SUN P.Research Progress in SoftwareTesting[J].Acta Scientiarum Naturalium Universitatis Peki-nensis,2005(1):134-145.
[5] BIRD D L,MUNOZC U.Automatic generation of random self-checking test cases[J].IBM systems journal,1983,22(3):229-245.
[6] CHEN T Y,LEUNG H,MAK I K.Adaptive random testing[C]//Annual Asian Computing Science Conference.Berlin:Springer,2004:320-329.
[7] SAHOO R R,RAY M.PSO based test case generation for criti-cal path using improved combined fitness function[J].Journal of King Saud University-Computer and Information Sciences,2020,32(4):479-490.
[8] BIDGOLI A M,HAGHIGHI H.Augmenting ant colony optimi-zation with adaptive random testing to cover prime paths[J].Journal of Systems and Software,2020,161:110495.
[9] YUAN Z,LOU Y,LIU M,et al.No more manual tests? evalua-ting and improving chatgpt for unit test generation[J].arXiv:2305.04207,2023.
[10] LI T O,ZONG W,WANG Y,et al.Nuances are the key:Unloc-king chatgpt to find failure-inducing tests with differential prompting[C]//2023 38th IEEE/ACM International Confe-rence on Automated Software Engineering(ASE).IEEE,2023:14-26.
[11] KARANJAI R,HUSSAIN A,RABIN M R I,et al.Harnessing the power of llms:Automating unit test generation for high-performance computing[J].arXiv:2407.05202,2024.
[12] UDUM U,KUMAR R,VASUMATHI D.Software Defect Prediction Using Temporal Transformer Graph Network with Newton-Raphson Optimization[J].Journal of Artificial Intelligence and Technology,2025(5):441-452.
[13] HE Y,HUANG J,RONG Y,et al.Unitsyn:A large-scale dataset capable of enhancing the prowess of large language models for program testing[C]//Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis.2024:1061-1072.
[14] JAIN K,SYNNAEVE G,ROZIÈRE B.Testgeneval:A realworld unit test generation and test completion benchmark[J].arXiv:2410.00752,2024.
[15] TUFANO M,DENG S K,SUNDARESAN N,et al.MethodsTest:A dataset of focal methods mapped to test cases[C]//Proceedings of the 19th International Conference on Mining Software Repositories.2022:299-303.
[16] LI B N,WANG B,CHEN M D,et al.Unit Test Generation for CPrograms via Large Language Models[J].Aerospace Contrd and Application,2025,51(1):115-122.
[17] JOHNSON L A.DO-178B:Software considerations in airbornesystems and equipment certification[J].Crosstalk,October,1998,199:11-20.
[18] SIAMI NAMIN A,ANDREWS J H,MURDOCHD J.Sufficient mutation operators for measuring test effectiveness[C]//Proceedings of the 30th International Conference on Software Engineering.2008:351-360.
[19] National Standardization Administration.Systems and software engineering—Software testing-Part 2:Test Processes:GB/T 38634.2-2020 [S].Beijing:Standards Press of China,2020.
[20] WEI J,WANG X,SCHUURMANS D,et al.Chain-of-thoughtprompting elicits reasoning in large language models[J].Advances in Neural Information Processing Systems,2022,35:24824-24837.
[21] WHITE J,FU Q,HAYS S,et al.A prompt pattern catalog toenhance prompt engineering with chatgpt[J].arXiv:2302.11382,2023.
[22] LIU A,FENG B,XUE B,et al.Deepseek-v3 technical report[J].arXiv:2412.19437,2024.
[23] WANG B.Research on Evaluation Index System and Evaluation Method of Software Test Cases [J].Ship Electronic Enginee-ring,2020,40(11):112-115.
[24] LIZ Z,LI H J.Research on dataset quality assessment model and method for large model training[J].Digital Economy,2024(12):26-30.
[25] YANG A,LI A,YANG B,et al.Qwen3 technical report[J].arXiv:2505.09388,2025.
[26] HOU X,ZHAO Y,WANG S,et al.Model context protocol(mcp):Landscape,security threats,and future research directions[J].arXiv:2503.23278,2025.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
No Suggested Reading articles found!