清单工程赋能多语言大语言模型评判器
Checklist Engineering Empowers Multilingual LLM Judges
AI总结：
针对多语言 LLM 评判依赖专有模型或微调数据的问题，本文提出无需训练的 CE-Judge 框架，利用清单工程和开源模型，在多个基准上达到优于基线且媲美 GPT-4o 的效果。
摘要：
自动化文本评估长期以来一直是自然语言处理（NLP）中的核心问题。近来，该领域转向使用大语言模型（LLM）作为评估者，这一趋势被称为 LLM-as-a-Judge 范式。尽管该方法前景良好且易于跨任务适配，但在多语言场景中的探索仍然有限。现有多语言研究通常依赖专有模型，或需要大量训练数据进行微调，从而引发成本、时间和效率方面的担忧。本文提出基于清单工程的 LLM-as-a-Judge（CE-Judge），这是一种无需训练的框架，利用清单直觉并结合开源模型进行多语言评估。在多种语言和三个基准数据集上、逐点与成对两种设置下的实验表明，我们的方法总体上优于基线，并取得与 GPT-4o 模型相当的表现。
英文摘要：
Automated text evaluation has long been a central issue in Natural Language Processing (NLP). Recently, the field has shifted toward using Large Language Models (LLMs) as evaluators-a trend known as the LLM-as-a-Judge paradigm. While promising and easily adaptable across tasks, this approach has seen limited exploration in multilingual contexts. Existing multilingual studies often rely on proprietary models or require extensive training data for fine-tuning, raising concerns about cost, time, and efficiency. In this paper, we propose Checklist Engineering based LLM-as-a-Judge (CE-Judge), a training-free framework that uses checklist intuition for multilingual evaluation with an open-source model. Experiments across multiple languages and three benchmark datasets, under both pointwise and pairwise settings, show that our method generally surpasses the baselines and performs on par with the GPT-4o model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
