发表机构
Mila - Quebec AI Institute & McGill University（米拉 - 魁北克人工智能研究所和麦吉尔大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究通过创建首个专家注释的含义取消数据集，评估大语言模型识别和理解潜在信念及更新的能力，发现其落后于人类，成功可能源于依赖先验信念，失败取决于类型和形式，表明当前LLMs未达人类水平。
AI 中文摘要
人类语言受潜在信念和信念更新驱动，这对大语言模型（LLMs）与用户成功交流至关重要。本文评估LLMs识别通过含义产生的潜在信念及通过含义取消理解其更新的能力，即话语隐含意义被削弱或否定的语用现象。创建首个专家注释的含义取消数据集，通过众包进行人类对含义及其相应取消的判断。发现LLMs信念更新理解落后于人类，尤其在更自然的场景中。额外控制实验表明，LLMs信念更新成功可能部分源于对先验信念的依赖，信念更新失败可能取决于其类型和形式。总体而言，当前LLMs尚未达到人类对潜在信念和信念更新的理解水平。
英文摘要
Human language is driven by unspoken beliefs and belief updates, making these critical to model for successful communication between large language models (LLMs) and their users. In this paper, we evaluate the ability of LLMs to recognize unspoken beliefs made through implicatures and to understand their updates through implicature cancellation: the pragmatic phenomenon whereby an utterance's implied meaning is weakened or negated. We create the first expert-annotated implicature cancellation dataset, ImplicatureX, crowdsourced for human judgements of implicatures and their corresponding cancellations. We find that LLM belief update understanding lags behind that of humans, especially in more naturally-occurring scenarios. Additional control experiments suggest that successes in LLM belief updates may stem in part from a reliance on prior beliefs, and that failures in belief updates may depend on their type and on their form. Overall, our study suggests that current LLMs have not yet reached human-level understanding of unspoken beliefs and belief updates. Code and data are available at https://github.com/cesare-spinoso/ImplicatureX.
