LLMs比黑猩猩更聪明吗?对视角取和知识状态估计的评估
Are LLMs Smarter Than Chimpanzees? An Evaluation on Perspective Taking and Knowledge State Estimation
AI总结:
本文评估LLM在预测角色下一步行动及检测角色不应具备的知识能力,发现当前最先进的LLM在两项任务上表现接近随机,远低于人类。
摘要:
认知人类学认为,人类智能的区别在于推断他人知识状态和理解意图的能力。相比之下,我们的近亲黑猩猩缺乏这种能力。本文旨在评估LLM在估计他人知识状态及其潜在行动方面的表现。我们设计了两个任务:(1)测试LLM是否能根据自身知识预测故事角色的下一步行动,还是错误使用不可见信息;(2)测试LLM是否能检测故事角色通过行动展示其不应具备的知识。结果表明,当前最先进的LLM在两项任务上的表现接近随机,且远低于人类。我们主张未来LLM研究应更加重视知识估计和意图理解能力。
英文摘要:
Cognitive anthropology suggests that the distinction of human intelligence lies in the ability to infer other individuals' knowledge states and understand their intentions. In comparison, our closest animal relative, chimpanzees, lack the capacity to do so. With this paper, we aim to evaluate LLM performance in estimating other individuals' knowledge states and their potential actions. We design two tasks to test (1) if LLMs can predict story characters' next actions based on their own knowledge vs. improperly using information unavailable from their perspective, and (2) if LLMs can detect when story characters, through their actions, demonstrate knowledge they should not possess. Results reveal that most current state-of-the-art LLMs achieve near-random performance on both tasks, and are substantially inferior to humans. We argue future LLM research should place more weight on the abilities of knowledge estimation and intention understanding.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。