复古代码，现代裁判：低数据体制下的元验证
Vintage Code, Modern Judges: Meta-Validation in Low Data Regimes
AI总结：
针对遗留语言应用现代化中高质量评估数据匮乏的问题，提出形式化框架SparseAlign，结合成对置信度与分数敏感的对齐度量，在稀疏人类标注数据下可靠评估并选择大模型裁判，已成功应用于COBOL代码解释的模型发布决策。
摘要：
诸如COBOL、PL/I和REXX等遗留语言的应用现代化面临着资源严重短缺的问题，无论是在专家可用性还是在高质量人类评估数据方面均如此。虽然将大型语言模型作为裁判（LaaJ）为专家评审提供了一种可扩展的替代方案，但在将其信任并应用于高风险工作流之前，必须验证其可靠性。若无原则性的验证，组织将面临循环评估回路的风险，即使用未经验证的LaaJ来评估模型输出，这可能会强化不可靠的判断并危及下游部署决策。尽管已有多种验证LaaJ的自动化方法被提出，但与人类判断的对齐仍是一种广泛使用且概念上具基础性的验证策略。在许多真实世界领域中，人类标注评估数据的可用性严重受限，使得难以评估LaaJ与人类判断的对齐程度。我们引入了SparseAlign，这是一个用于在稀疏人类标注数据下评估LaaJ对齐情况的形式化框架。SparseAlign结合了一种新颖的成对置信度概念与一种分数敏感的对齐度量指标，二者共同捕捉排名一致性与分数接近度，即使在传统统计方法因标注样本有限而失效时，也能实现可靠的评估器选择。SparseAlign被内部应用于为COBOL代码解释选择LaaJ。对齐度最高的评估器被集成到评估工作流中，用于指导模型发布决策。我们展示了一个包含四个LaaJ的案例研究，以证明SparseAlign在真实评估场景中的效用。
英文摘要：
Application modernization in legacy languages such as COBOL, PL/I, and REXX faces an acute shortage of resources, both in expert availability and in high-quality human evaluation data. While Large Language Models as a Judge (LaaJ) offer a scalable alternative to expert review, their reliability must be validated before being trusted in high-stakes workflows. Without principled validation, organizations risk a circular evaluation loop, where unverified LaaJs are used to assess model outputs, potentially reinforcing unreliable judgments and compromising downstream deployment decisions. Although various automated approaches to validating LaaJs have been proposed, alignment with human judgment remains a widely used and conceptually grounded validation strategy. In many real-world domains, the availability of human-labeled evaluation data is severely limited, making it difficult to assess how well a LaaJ aligns with human judgment. We introduce SparseAlign, a formal framework for assessing LaaJ alignment with sparse human-labeled data. SparseAlign combines a novel pairwise-confidence concept with a score-sensitive alignment metric that jointly capture ranking consistency and score proximity, enabling reliable evaluator selection even when traditional statistical methods are ineffective due to limited annotated examples. SparseAlign was applied internally to select LaaJs for COBOL code explanation. The top-aligned evaluators were integrated into assessment workflows, guiding model release decisions. We present a case study of four LaaJs to demonstrate SparseAlign's utility in real-world evaluation scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
