arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.05083cs.CLcs.AIcs.LG
收藏

超越LLM作为裁判:多语言生成文本评估的确定性度量

Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation

Qatar Computing Research Institute, HBKU, Qatar(卡塔尔哈马德·本·哈利法大学卡塔尔计算研究所);UPES, India(印度UPES大学)

Firoj Alam, Gagan Bhatia, Sahinur Rahman Laskar, Shammur Absar Chowdhury

AI总结:

本文提出OmniScore,一种基于小参数模型的确定性度量,用于多语言生成文本评估,提供低延迟和一致性的评分方法,通过大规模合成监督训练,验证了其在问答、翻译和总结任务中的有效性。

摘要:

尽管大型语言模型(LLM)被越来越多地用作自动裁判来评估生成文本,但其输出成本高且对提示设计、语言和聚合策略敏感,严重影响可重复性。为解决这些挑战,我们提出了OmniScore,一种使用小规模(<1B参数)模型开发的互补、确定性学习度量。OmniScore近似LLM裁判行为,同时保持传统模型评分的低延迟和一致性。我们通过大规模合成监督(约564k实例,在107种语言中)训练模型,并使用8,617个手动标注实例进行评估。OmniScore家族支持在多种设置中提供可靠的多维评分,包括基于参考、源地基和混合评估。我们在6种语言中评估这些模型在问答、翻译和总结任务中的表现。我们的结果表明,轻量级、确定性学习度量为前沿LLM提供了一种高度实用且可扩展的替代方案。我们的模型和数据集可在https://huggingface.co/collections/QCRI/omniscore找到。

英文摘要:

While Large Language Models (LLMs) are increasingly adopted as automated judges for evaluating generated text, their outputs are often costly, and highly sensitive to prompt design, language, and aggregation strategies, severely, which limits reproducibility. To address these challenges, we propose \textbf{\textit{OmniScore}}, a family of complementary, deterministic learned metrics developed using small size ($<$1B) parameter models. OmniScore approximates LLM-judge behavior while preserving the low latency and consistency of traditional model-based scoring. We trained the models large-scale synthetic supervision ($\sim$564k instances, in \textbf{107 languages}) and evaluated using 8,617 manually annotated instances. The OmniScore family supports reliable, multi-dimensional scores across a variety of settings, including reference-based, source-grounded, and hybrid evaluations. We evaluate these models across question answering (QA), translation, and summarization in \textbf{6 languages}. Our results demonstrate that lightweight, deterministic learned metrics provide a highly practical and scalable alternative to frontier LLMs. Our models and datasets can be found at https://huggingface.co/collections/QCRI/omniscore

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑