arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.02451cs.CL
收藏

观点属于我,也属于你:利用共同认知基准评估心智理论

Views Are My Own, but Also Yours: Benchmarking Theory of Mind Using Common Ground

Department of Computer Science(计算机科学系);Department of Linguistics(语言学系);Institute for Advanced Computational Science(高级计算科学研究院);Stony Brook University(石溪大学);University of Guilan(吉兰大学)

Adil Soubki, John Murzaku, Arash Yousefi Jordehi, Peter Zeng, Magdalena Markowska, Seyed Abolghasem Mirroshandel, Owen Rambow

AI总结:

本文提出首个基于自然口语对话的心智理论基准Common-ToM,发现语言模型表现不佳,并证明引入显式信念表示可提升其ToM能力。

摘要:

评估语言模型(LMs)的心智理论(ToM)能力近来受到大量关注。然而,许多现有基准依赖合成数据,这可能导致实验与人类行为不一致。我们提出了首个基于自然发生的口语对话的ToM数据集Common-ToM,并表明语言模型在展示ToM方面存在困难。随后,我们证明引入一种简单、显式的信念表示能够提升语言模型在Common-ToM上的表现。

英文摘要:

Evaluating the theory of mind (ToM) capabilities of language models (LMs) has recently received a great deal of attention. However, many existing benchmarks rely on synthetic data, which risks misaligning the resulting experiments with human behavior. We introduce the first ToM dataset based on naturally occurring spoken dialogs, Common-ToM, and show that LMs struggle to demonstrate ToM. We then show that integrating a simple, explicit representation of beliefs improves LM performance on Common-ToM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑