基于LLM的软件工程工件评估器的自动化验证
Automated Validation of LLM-based Evaluators for Software Engineering Artifacts
AI总结：
针对LLM评估器可靠性挑战，提出REFINE框架，通过合成降级工件和排名对齐测试自动验证评估器，在COBOL任务中将一致性得分从0.7提升至0.9以上。
摘要：
软件工程中的自动化日益依赖大型语言模型（LLMs）来生成、审查和评估代码工件。然而，将LLMs确立为可靠的评估器仍是一个开放挑战：人工评估成本高昂、主观且不可扩展，而现有的自动化方法无法辨别工件质量的细微差异。我们引入了REFINE（Ranking Evaluators for FIne grained Nuanced Evaluation），一个用于跨软件工程任务对基于LLM的评估器进行基准测试的自动化框架。REFINE包含两个模块：层级数据集构建器（Hierarchy Dataset Builder）应用新颖的生成技术自动合成质量逐步降低的工件，评估器测试器（Evaluator Tester）通过测量每个候选评估器配置的排名与预期顺序的吻合程度来量化其性能。REFINE的一个关键特性是可控性：用户可以调整降级的粒度，以逐步细化评估器配置，从粗粒度过滤到对细微质量差距的压力测试。尽管该方法具有通用性，我们聚焦于编码任务，反映了生产环境中的实际需求。REFINE已集成到IBM的内部开发工作流中，并应用于COBOL（一种企业关键编程语言）的代码生成、翻译和摘要任务，使用了工业数据。它被用于识别LLM作为评判者的配置，在某些编码任务中将一致性得分从低于0.7提升到0.9以上。这些对细微差别敏感的评估器现已被模型训练团队积极使用，以支持模型发布决策。
英文摘要：
Automation in software engineering increasingly relies on large language models (LLMs) to generate, review, and assess code artifacts. However, establishing LLMs as reliable evaluators remains an open challenge: human evaluations are costly, subjective and non scalable, while existing automated methods fail to discern fine grained variations in artifact quality. We introduce REFINE (Ranking Evaluators for FIne grained Nuanced Evaluation), an automated framework for benchmarking LLM based evaluators across software engineering tasks. REFINE comprises of two modules: Hierarchy Dataset Builder applies novel generation techniques to automatically synthesize artifacts with progressively reduced quality, and Evaluator Tester quantifies each candidate evaluator configuration by measuring how closely its rankings align with expected ordering. A key feature of REFINE is controllability: users can tune the granularity of degradation to progressively refine evaluator configurations, from coarse filtering to stress testing on subtle quality gaps. While the methodology is general, we focus on coding tasks reflecting the practical demands in our production setting. REFINE was integrated into IBM's internal development workflows and applied to code generation, translation, and summarization for COBOL, an enterprise critical programming language, using industrial data. It was used to identify LLM as a Judge configurations that lifted alignment scores from below $0.7$ to above $0.9$ in some coding tasks. These nuance sensitive evaluators are now actively used by model training teams to support model release decisions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
