LLM-Rubric：一种面向自然语言文本自动评估的多维校准方法
LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts
AI总结：
本文提出LLM-Rubric框架，通过训练小型前馈神经网络校准多个LLM对评分标准问题的回答分布，以预测人类评判者的多维标注，在对话系统评估中将用户满意度预测误差降至0.5以下，较未校准基线提升2倍。
摘要：
本文介绍了一个用于自然语言文本自动评估的框架。一个人工构建的评分标准描述了如何评估多个感兴趣的维度。为了评估一段文本，一个大型语言模型（LLM）被提示回答每个评分标准问题，并生成一个关于潜在回答的分布。LLM的预测往往与人类评判者的判断不太一致——事实上，人类评判者之间也并非完全一致。然而，多个LLM分布可以被组合起来，以预测每位人类评判者在所有问题上的标注，包括一个评估整体质量或相关性的总结性问题。LLM-Rubric通过训练一个小型前馈神经网络来实现这一点，该网络同时包含评判者特定参数和评判者无关参数。在一个人工智能信息寻求任务中评估对话系统时，我们发现，使用9个问题（评估自然度、简洁性和引用质量等维度）的LLM-Rubric能够预测人类评判者对整体用户满意度的评估（评分范围为1到4），其RMS误差小于0.5，比未校准基线提高了2倍。
英文摘要：
This paper introduces a framework for the automated evaluation of natural language texts. A manually constructed rubric describes how to assess multiple dimensions of interest. To evaluate a text, a large language model (LLM) is prompted with each rubric question and produces a distribution over potential responses. The LLM predictions often fail to agree well with human judges -- indeed, the humans do not fully agree with one another. However, the multiple LLM distributions can be $\textit{combined}$ to $\textit{predict}$ each human judge's annotations on all questions, including a summary question that assesses overall quality or relevance. LLM-Rubric accomplishes this by training a small feed-forward neural network that includes both judge-specific and judge-independent parameters. When evaluating dialogue systems in a human-AI information-seeking task, we find that LLM-Rubric with 9 questions (assessing dimensions such as naturalness, conciseness, and citation quality) predicts human judges' assessment of overall user satisfaction, on a scale of 1--4, with RMS error $< 0.5$, a $2\times$ improvement over the uncalibrated baseline.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
