TestDecision: 通过贪婪优化和强化学习进行序列测试套件生成
TestDecision: Sequential Test Suite Generation via Greedy Optimization and Reinforcement Learning
AI总结:
本文提出TestDecision,通过将LLM转化为神经贪婪专家,利用贪婪策略和强化学习生成高质量测试套件,显著提升分支覆盖率和执行通过率。
摘要:
随着LLM的快速发展,自动化软件测试正经历范式转变。尽管专有模型如GPT-4o表现出色,但其高部署成本和数据隐私问题使开源LLM成为许多学术和工业场景的现实需求。在自动化测试生成领域,已发展为基于LLM的迭代工作流来构建测试套件。在使用开源LLM时,我们实证发现它们缺乏套件级视角,存在结构性短视问题,无法基于当前覆盖状态生成具有大边际收益的新测试。本文从序列角度出发,将测试套件生成形式化为MDP,并证明其目标表现出单调次模特性,这使NP难的全局优化问题可以有效放松为可处理的逐步贪婪过程。基于此洞察,我们提出TestDecision,将LLM转化为神经贪婪专家。TestDecision包含两个协同组件:(1)一个实现逐步贪婪策略的推理框架;(2)一个强化学习训练流程,使基础LLM具备序列测试生成能力以最大化边际收益。在ULT基准的全面评估中,TestDecision显著优于现有先进方法。它在分支覆盖率和执行通过率上比所有基础模型提高38.15-52.37%和298.22-558.88%,在7B基础模型上表现与专有LLM GPT-5.2相当。此外,TestDecision能发现比普通基础LLM多58.43-95.45%的bug,并在LiveCodeBench上表现出更优的泛化能力,证明其能够构建高质量的测试套件。
英文摘要:
With the rapid evolution of LLMs, automated software testing is witnessing a paradigm shift. While proprietary models like GPT-4o demonstrate impressive capabilities, their high deployment costs and data privacy concerns make open-source LLMs the practical imperative for many academic and industrial scenarios. In the field of automated test generation, it has evolved to iterative workflows to construct test suites based on LLMs. When utilizing open-source LLMs, we empirically observe they lack a suite-level perspective, suffering from structural myopia-failing to generate new tests with large marginal gain based on the current covered status. In this paper, from the perspective of sequences, we formalize test suite generation as a MDP and demonstrate that its objective exhibits monotone submodularity, which enables an effective relaxation of this NP-hard global optimization into a tractable step-wise greedy procedure. Guided by this insight, we propose TestDecision, which transforms LLMs into neural greedy experts. TestDecision consists of two synergistic components: (1) an inference framework which implements test suite construction following a step-wise greedy strategy; and (2) a training pipeline of reinforcement learning which equips the base LLM with sequential test generation ability to maximize marginal gain. Comprehensive evaluations on the ULT benchmark demonstrate that TestDecision significantly outperforms existing advanced methods. It brings an improvement between 38.15-52.37% in branch coverage and 298.22-558.88% in execution pass rate over all base models, achieving a comparable performance on 7B backbone with a much larger proprietary LLM GPT-5.2. Furthermore, TestDecision can find 58.43-95.45% more bugs than vanilla base LLMs and exhibit superior generalization on LiveCodeBench, proving its capability to construct high-quality test suites.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。