AXIOM：通过基于规则的扰动和多源质量校准构建LLM-as-a-Judge代码评估基准
AXIOM: Benchmarking LLM-as-a-Judge for Code via Rule-Based Perturbation and Multisource Quality Calibration
AI总结：
AXIOM通过基于规则的扰动和多源质量校准构建代码评估基准，以实现高质量代码评分的平衡分布和可靠评估。
摘要：
大型语言模型（LLM）在现实软件工程中的应用日益增加，推动了代码评估指标的发展，以研究LLM生成代码的质量。传统基于规则的指标仅根据参考程序的表层相似性对程序进行评分，而没有深入分析功能和代码质量。为解决这一限制，研究者开发了LLM-as-a-judge指标，促使LLM评估和评分代码，并构建了各种代码评估基准以验证其有效性。然而，这些基准存在关键限制，阻碍了评估能力的可靠评估：一些特征具有粗粒度的二元标签，将丰富的代码行为简化为单一比特信息，掩盖了细微错误。其他则提出细粒度但主观、模糊定义的评估标准，引入了依赖的地面真相的不可靠性。此外，它们经常使用不受控的数据合成方法，导致评分分布不平衡，无法代表现实世界的代码生成场景。为了构建一个具有各种质量水平平衡分布的多样化基准，并简化手动注释流程，我们提出AXIOM，一种新的基于扰动的框架，用于大规模合成代码评估基准。它将程序评分重新定义为部署所需的细化工作量，包括两个阶段：（1）规则引导的扰动，促使LLM应用一系列预定义的扰动规则到现有高质量程序上，以修改其功能和代码质量，使我们能够精确控制每个程序的目标评分以实现评分分布的平衡。（2）多源质量校准，首先选择一个子集...
英文摘要：
Large language models (LLMs) have been increasingly deployed in real-world software engineering, fostering the development of code evaluation metrics to study the quality of LLM-generated code. Conventional rule-based metrics merely score programs based on their surface-level similarities with reference programs instead of analyzing functionality and code quality in depth. To address this limitation, researchers have developed LLM-as-a-judge metrics, prompting LLMs to evaluate and score code, and curated various code evaluation benchmarks to validate their effectiveness. However, these benchmarks suffer from critical limitations, hindering reliable assessments of evaluation capability: Some feature coarse-grained binary labels, which reduce rich code behavior to a single bit of information, obscuring subtle errors. Others propose fine-grained but subjective, vaguely-defined evaluation criteria, introducing unreliability in manually-annotated scores, which is the ground-truth they rely on. Furthermore, they often use uncontrolled data synthesis methods, leading to unbalanced score distributions that poorly represent real-world code generation scenarios. To curate a diverse benchmark with programs of well-balanced distributions across various quality levels and streamline the manual annotation procedure, we propose AXIOM, a novel perturbation-based framework for synthesizing code evaluation benchmarks at scale. It reframes program scores as the refinement effort needed for deployment, consisting of two stages: (1) Rule-guided perturbation, which prompts LLMs to apply sequences of predefined perturbation rules to existing high-quality programs to modify their functionality and code quality, enabling us to precisely control each program's target score to achieve balanced score distributions. (2) Multisource quality calibration, which first selects a subset of...
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
