PUB：用于评估大语言模型语用能力的语用理解基准
PUB: A Pragmatics Understanding Benchmark for Assessing LLMs' Pragmatics Capabilities
AI总结：
本文发布包含四类语用现象、十四项任务的 PUB 基准，通过 28k 数据点评估多种 LLM，发现指令微调对小型模型提升显著，而大型模型基础版与对话版相当，且模型能力与人类仍有明显差距。
摘要：
大语言模型（LLMs）在理解语义方面已展现出卓越能力，但在理解语用方面往往存在困难。为证明这一事实，我们发布了语用理解基准（PUB）数据集，该数据集包含四个语用现象中的十四项任务，即：含义（Implicature）、预设（Presupposition）、指称（Reference）和指示（Deixis）。我们为每项任务精心整理了高质量的测试集，由多项选择问答（MCQA）组成。PUB 共包含 28k 个数据点，其中 6.1k 由我们创建，其余改编自现有数据集。我们评估了九个在参数数量和训练类型上各不相同的模型。研究表明，针对指令遵循和对话的微调显著增强了较小语言模型的语用能力。然而，对于较大的模型，基础版本与经过对话适配的对应版本表现相当。此外，人类能力与模型能力之间存在明显的性能差距。并且，与人类在各种任务中表现一致不同，模型展示出其熟练程度的差异性，性能水平因同一数据集内不同提示和任务复杂性而波动。总体而言，该基准旨在对 LLM 处理需要语用推理的真实世界语言任务的能力提供全面评估。
英文摘要：
LLMs have demonstrated remarkable capability for understanding semantics, but they often struggle with understanding pragmatics. To demonstrate this fact, we release a Pragmatics Understanding Benchmark (PUB) dataset consisting of fourteen tasks in four pragmatics phenomena, namely, Implicature, Presupposition, Reference, and Deixis. We curated high-quality test sets for each task, consisting of Multiple Choice Question Answers (MCQA). PUB includes a total of 28k data points, 6.1k of which have been created by us, and the rest are adapted from existing datasets. We evaluated nine models varying in the number of parameters and type of training. Our study indicates that fine-tuning for instruction-following and chat significantly enhances the pragmatics capabilities of smaller language models. However, for larger models, the base versions perform comparably with their chat-adapted counterparts. Additionally, there is a noticeable performance gap between human capabilities and model capabilities. Furthermore, unlike the consistent performance of humans across various tasks, the models demonstrate variability in their proficiency, with performance levels fluctuating due to different hints and the complexities of tasks within the same dataset. Overall, the benchmark aims to provide a comprehensive evaluation of LLM's ability to handle real-world language tasks that require pragmatic reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
