面向可扩展的可靠自动化评估：基于大语言模型
(Towards) Scalable Reliable Automated Evaluation with Large Language Models
浏览论文内容
中文总结 AI 辅助
本研究提出一种基于多LLM两两比较与Elo评分的自动化评估框架，可近似专家评估，减少人工干预，实现对LLM输出的高效可靠评估。
中文摘要 AI 辅助
评估大语言模型（LLMs）生成的文本输出的质量与相关性，仍是一项具有挑战性且资源密集型的任务。现有自动化指标往往无法捕捉LLM生成输出固有的复杂性与变异性，且这些指标通常依赖明确的参考标准，限制了其在仅具备客观基准的领域中的应用。本研究提出一种新颖的评估框架，旨在近似专家对LLM生成内容的评估。该方法采用多个LLM对输出进行两两比较，以减少单一模型带来的偏差；使用Elo评分系统生成稳定且可解释的排名；设置从全一致到多数投票的可调一致阈值，可灵活控制评估的置信度与覆盖范围。通过评估从科学摘要中提取的能力轮廓，验证了该方法的有效性，初步结果显示自动生成的排名与专家判断具有良好的相关性，大幅减少了对大量人工干预的需求。该框架提供了可扩展、一致且领域无关的评估层，支持在各类应用中对LLM输出进行更高效、可靠的质量评估。
英文摘要
Evaluating the quality and relevance of textual outputs from Large Language Models (LLMs) remains challenging and resource-intensive. Existing automated metrics often fail to capture the complexity and variability inherent in LLM-generated outputs. Moreover, these metrics typically rely on explicit reference standards, limiting their use mostly to domains with objective benchmarks. This work introduces a novel evaluation framework designed to approximate expert-level assessments of LLM-generated content. The proposed method employs pairwise comparisons of outputs by multiple LLMs, reducing biases from individual models. An Elo rating system is used to generate stable and interpretable rankings. Adjustable agreement thresholds, from full unanimity to majority voting, allow flexible control over evaluation confidence and coverage. The method's effectiveness is demonstrated through evaluating competency profiles extracted from scientific abstracts. Preliminary results show that automatically derived rankings correlate well with expert judgments, significantly reducing the need for extensive human intervention. By offering a scalable, consistent, and domain-agnostic evaluation layer, the framework supports more efficient and reliable quality assessments of LLM outputs across diverse applications.
发表机构
- KIT（卡尔斯鲁厄理工学院）
机构由 AI 辅助整理，请以论文原文为准。
