神谕已言：Pythia 对话能力的多维度评估
The Oracle Has Spoken: A Multi-Aspect Evaluation of Dialogue in Pythia
AI总结：
本文通过多维度细粒度指标评估 Pythia 模型对话能力，发现模型规模影响有限而微调迅速饱和，并揭示基于同一评估器的指标存在可靠性问题。
摘要：
对话是大语言模型（LLMs）的标志性能力之一。尽管对话无处不在，但很少有研究真正区分在后训练过程中支撑对话行为出现的具体要素。我们采用了一套全面的基于模型的评估指标，每个指标都以语言学理论为依据，针对对话中一个不同的细粒度方面。我们评估了预训练 Pythia 模型在这些维度上的表现如何随模型规模变化，以及在对话数据集上进行监督微调后的结果。我们观察到，原始模型规模对大多数指标只有轻微影响，而微调会迅速使除最小模型外的所有模型的得分趋于饱和。与我们的预期有些相反，许多指标表现出非常相似的趋势，尤其是当它们都植根于同一个评估器模型时，这引发了对其在衡量特定维度方面可靠性的质疑。为此，我们对生成响应中的分数分布、指标相关性和词项频率进行了额外分析，以帮助解释我们的观察结果。
英文摘要：
Dialogue is one of the landmark abilities of large language models (LLMs). Despite its ubiquity, few studies actually distinguish specific ingredients underpinning dialogue behavior emerging during post-training. We employ a comprehensive suite of model-based metrics, each targeting a distinct fine-grained aspect of dialogue, motivated by linguistic theory. We evaluate how the performance of pre-trained Pythia models changes with respect to each of those dimensions, depending on model size and as a result of supervised fine-tuning on conversational datasets. We observe only a mild impact of raw model size on most metrics, whereas fine-tuning quickly saturates the scores for all but the smallest models tested. Somewhat contrary to our expectations, many metrics show very similar trends, especially if they are all rooted in the same evaluator model, which raises the question of their reliability in measuring a specific dimension. To that end, we conduct additional analyses of score distributions, metric correlations, and term frequencies in generated responses to help explain our observations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
