arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.01232cs.SEcs.AI
收藏

基于LLM的判别器用于可扩展的测试覆盖率评估:准确性、操作可靠性与成本

LLM-as-a-Judge for Scalable Test Coverage Evaluation: Accuracy, Operational Reliability, and Cost

School of Computing and Information Systems(计算与信息系统学院);Singapore Management University(新加坡管理大学);Research and Development(研发)

Donghao Huang, Shila Chew, Anna Dutkiewicz, Zhaoxia Wang

AI总结:

LLM-as-a-Judge通过低成本高可靠性的测试覆盖率评估方法,显著降低评估成本并提升准确性。

摘要:

在质量保证流水线中,大规模评估软件测试覆盖率仍然是一个瓶颈。我们提出了LLM-as-a-Judge(LAJ),一个生产就绪的、基于评分标准的框架,用于评估Gherkin接受测试,并生成结构化的JSON输出。在20种模型配置(GPT-4、GPT-5以及不同推理努力的开放权重模型)上,对100个专家注释的脚本进行了5次运行(500次评估),我们提供了首次全面的分析,涵盖准确性、操作可靠性和成本。我们引入了评估完成率(ECR@1)来量化首次尝试的成功率,揭示了可靠性从85.4%到100.0%,通过重试带来了实质性成本影响。结果表明,较小的模型可以超越较大的模型:GPT-4o Mini在准确性(6.07 MAAE)、高可靠性(96.6% ECR@1)和低成本(每1000次1.01美元)方面表现最佳,相比GPT-5(高推理)成本降低了78倍。推理努力是模型家族依赖的:GPT-5从增加推理中受益(有可预测的准确性-成本权衡),而开放权重模型在所有维度上随着推理增加而下降。总体而言,成本跨度达175倍(每1000次0.45-78.96美元)。我们发布了数据集、框架和代码以支持可重复性和部署。

英文摘要:

Assessing software test coverage at scale remains a bottleneck in QA pipelines. We present LLM-as-a-Judge (LAJ), a production-ready, rubric-driven framework for evaluating Gherkin acceptance tests with structured JSON outputs. Across 20 model configurations (GPT-4, GPT-5 with varying reasoning effort, and open-weight models) on 100 expert-annotated scripts over 5 runs (500 evaluations), we provide the first comprehensive analysis spanning accuracy, operational reliability, and cost. We introduce the Evaluation Completion Rate (ECR@1) to quantify first-attempt success, revealing reliability from 85.4% to 100.0% with material cost implications via retries. Results show that smaller models can outperform larger ones: GPT-4o Mini attains the best accuracy (6.07 MAAE), high reliability (96.6% ECR@1), and low cost ($1.01 per 1K), yielding a 78x cost reduction vs. GPT-5 (high reasoning) while improving accuracy. Reasoning effort is model-family dependent: GPT-5 benefits from increased reasoning (with predictable accuracy-cost tradeoffs), whereas open-weight models degrade across all dimensions as reasoning increases. Overall, cost spans 175x ($0.45-$78.96 per 1K). We release the dataset, framework, and code to support reproducibility and deployment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑