HumanRankEval：作为对话助手的语言模型的自动评估
HumanRankEval: Automatic Evaluation of LMs as Conversational Assistants
AI总结：
针对对话助手类语言模型人工评估不可扩展、自动评估难衡量对话与指令遵循能力的问题，提出基于对数似然排序并与人类排序求相关的HumanRankEval评估任务，验证其与人类判断相关性好且对指令微调变化敏感。
摘要：
作为对话助手的语言模型（LMs）近来已成为帮助人们完成各类任务的热门工具。这类模型通常是对在通用领域文本序列上预训练的LM，通过进一步的指令微调以及可能的偏好优化方法适配而来。理想情况下，这类LM的评估应借助人类判断来开展，但这一方式不具备可扩展性。另一方面，以辅助LM作为评委和/或采用基于知识的任务的自动评估方式具备可扩展性，却难以评估对话能力和指令遵循度。为助力加快作为对话助手的LM的开发进程，我们提出了一项新颖的自动评估任务：HumanRankEval（HRE）。它包含一套大规模、多样化且高质量的问题集，每个问题都配有多个人类撰写并打分的答案。执行评估时，HRE基于这些答案在LM分布下的对数似然值对其进行排序，随后计算这些排序与对应的人类排序之间的相关性。我们通过研究HRE区分不同规模的预训练LM和指令微调LM的有效程度，验证了其效用。结果表明，HRE与人类判断具有良好的相关性，且对指令微调后模型的变化尤为敏感。
英文摘要：
Language models (LMs) as conversational assistants recently became popular tools that help people accomplish a variety of tasks. These typically result from adapting LMs pretrained on general domain text sequences through further instruction-tuning and possibly preference optimisation methods. The evaluation of such LMs would ideally be performed using human judgement, however, this is not scalable. On the other hand, automatic evaluation featuring auxiliary LMs as judges and/or knowledge-based tasks is scalable but struggles with assessing conversational ability and adherence to instructions. To help accelerate the development of LMs as conversational assistants, we propose a novel automatic evaluation task: HumanRankEval (HRE). It consists of a large-scale, diverse and high-quality set of questions, each with several answers authored and scored by humans. To perform evaluation, HRE ranks these answers based on their log-likelihood under the LM's distribution, and subsequently calculates their correlation with the corresponding human rankings. We support HRE's efficacy by investigating how efficiently it separates pretrained and instruction-tuned LMs of various sizes. We show that HRE correlates well with human judgements and is particularly responsive to model changes following instruction-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
