面向软件工程的LLM-as-a-Judge：文献综述、愿景与未来路线
LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead
AI总结：
本文针对软件工程中LLM生成制品评估缺乏可靠可扩展方法的问题，综述现有LLM-as-a-Judge相关研究、分析局限与空白，提出2030年发展路线图，以推动该范式的研究应用，提升评估可扩展性。
摘要：
大语言模型（Large Language Models, LLMs）在软件工程（software engineering, SE）中的快速集成革新了代码生成等任务，产出了海量软件制品。这一增长暴露出一个关键瓶颈：缺乏可扩展、可靠的方法来评估这些输出。人工评估成本高、耗时长，而BLEU等传统自动化指标无法捕捉质量的细微差异。为此，LLM-as-a-Judge（大语言模型作为评判者）范式——即利用LLM进行自动化评估——应运而生。该方法借助LLM的高级推理能力，为在自动化规模下实现类人类的细微评估提供了路径。然而，软件工程领域的LLM-as-a-Judge研究仍处于早期阶段。这篇具有前瞻性的SE 2030论文旨在引导学界推进LLM-as-a-Judge在LLM生成软件制品评估中的应用。我们对现有软件工程相关研究进行了文献综述，分析了其局限性，明确了关键研究空白，并勾勒出详细的路线图。我们设想，到2030年，这些框架将成为可靠、稳健且可扩展的人类替代者，能够对软件制品进行一致、多维度的评估。我们的工作旨在推动LLM-as-a-Judge框架的研究与应用，最终提升软件制品评估的可扩展性。
英文摘要：
The rapid integration of Large Language Models (LLMs) into software engineering (SE) has revolutionized tasks like code generation, producing a massive volume of software artifacts. This surge has exposed a critical bottleneck: the lack of scalable, reliable methods to evaluate these outputs. Human evaluation is costly and time-consuming, while traditional automated metrics like BLEU fail to capture nuanced quality aspects. In response, the LLM-as-a-Judge paradigm - using LLMs for automated evaluation - has emerged. This approach leverages the advanced reasoning of LLMs, offering a path toward human-like nuance at automated scale. However, LLM-as-a-Judge research in SE is still in its early stages. This forward-looking SE 2030 paper aims to steer the community toward advancing LLM-as-a-Judge for evaluating LLM-generated software artifacts. We provide a literature review of existing SE studies, analyze their limitations, identify key research gaps, and outline a detailed roadmap. We envision these frameworks as reliable, robust, and scalable human surrogates capable of consistent, multi-faceted artifact evaluation by 2030. Our work aims to foster research and adoption of LLM-as-a-Judge frameworks, ultimately improving the scalability of software artifact evaluation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
