大型语言模型在自然语言生成评估中是主动评判者
Large Language Models Are Active Critics in NLG Evaluation
AI总结：
本文提出Active-Critic评估器，通过自推断任务与标准并动态优化提示，使LLM能以少量示例适应多样NLG评估，并在多任务上更好地对齐人类判断。
摘要：
使用大型语言模型（LLM）进行自然语言生成（NLG）评估的传统范式依赖于预定义的任务定义和评估标准，使LLM处于“被动评判者”的位置，严格遵循开发者提供的指南。然而，人类评估者往往运用隐性标准，而且他们在实践中的期望会根据特定终端用户需求而有很大差异。因此，这些僵化的评估方法若不进行大量提示定制，便难以适应多样化场景。为解决这一问题，我们提出了Active-Critic，这是一种新颖的基于LLM的评估器，可将LLM转变为“主动评判者”，能够利用有限的示例数据适应多样化的NLG任务。Active-Critic包含两个阶段：（1）自推断目标NLG任务及相关评估标准；（2）动态优化提示，以生成与人类一致的分数以及详细的理由说明。我们的实验表明，Active-Critic能够生成细致入微、具备情境感知能力的评估标准，从而使其在多个任务上实现与人类判断更优的一致性。
英文摘要：
The conventional paradigm of using large language models (LLMs) for natural language generation (NLG) evaluation relies on pre-defined task definitions and evaluation criteria, positioning LLMs as "passive critics" that strictly follow developer-provided guidelines. However, human evaluators often apply implicit criteria, and their expectations in practice can vary widely based on specific end-user needs. Consequently, these rigid evaluation methods struggle to adapt to diverse scenarios without extensive prompt customization. To address this, we introduce Active-Critic, a novel LLM-based evaluator that transforms LLMs into "active critics'' capable of adapting to diverse NLG tasks using limited example data. Active-Critic consists of two stages: (1) self-inferring the target NLG task and relevant evaluation criteria, and (2) dynamically optimizing prompts to produce human-aligned scores along with detailed justifications. Our experiments show that Active-Critic can generate nuanced, context-aware evaluation criteria, enabling it to achieve superior alignment with human judgments across multiple tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
