arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.16085cs.CLcs.AI
收藏

语言统计与虚假信念推理:来自41个开放权重语言模型的证据

Language Statistics and False Belief Reasoning: Evidence from 41 Open-Weight LMs

Rutgers University - Newark(新泽西罗格斯大学-新布朗斯维尔);UC San Diego(加州大学圣地亚哥分校);Stony Brook University(史泰文斯理工学院);MIT(麻省理工学院)

Sean Trott, Samuel Taylor, Cameron Jones, James A. Michaelov, Pamela D. Rivière

AI总结:

该研究通过41个开放权重语言模型测试,发现人类和模型在非事实性动词提示下对虚假信念的归因存在偏差,揭示语言统计学在解释人类认知中的局限性。

摘要:

对语言模型(LMs)中的心理状态推理进行研究有潜力为人类社会认知理论(如心理状态推理部分源于语言暴露的理论)提供信息,并加深我们对LMs自身的理解。然而,许多发表的LMs研究依赖于相对较小的封闭源LMs样本,限制了我们严格测试心理理论和评估LM能力的能力。在此,我们通过在41个开放权重模型(来自不同模型家族)上评估LM的心理状态推理行为,复制并扩展了已发表的虚假信念任务研究。我们发现,在34%的测试LM中表现出对隐含知识状态的敏感性;然而,与先前研究一致,没有一个完全“解释掉”人类的效果。较大的LMs显示出更高的敏感性,并且也表现出更高的心理计量预测能力。最后,我们利用LM的行为生成并测试了一个关于人类认知的新假设:当使用非事实性动词(“John thinks...”)提示知识状态时,人类和LMs都倾向于倾向于归因虚假信念,而不是当使用间接提示(“John looks in the...”)时。与知识状态的主要效应不同,其中人类的敏感性超过LMs,人类知识提示效应的幅度正好落在LM效应大小的分布范围内——这表明语言统计学在原则上可以解释后者,但不能解释前者在人类中。这些结果证明了使用更大的开放权重LMs样本来测试人类认知理论和评估LM能力的价值。

英文摘要:

Research on mental state reasoning in language models (LMs) has the potential to inform theories of human social cognition--such as the theory that mental state reasoning emerges in part from language exposure--and our understanding of LMs themselves. Yet much published work on LMs relies on a relatively small sample of closed-source LMs, limiting our ability to rigorously test psychological theories and evaluate LM capacities. Here, we replicate and extend published work on the false belief task by assessing LM mental state reasoning behavior across 41 open-weight models (from distinct model families). We find sensitivity to implied knowledge states in 34% of the LMs tested; however, consistent with prior work, none fully ``explain away'' the effect in humans. Larger LMs show increased sensitivity and also exhibit higher psychometric predictive power. Finally, we use LM behavior to generate and test a novel hypothesis about human cognition: both humans and LMs show a bias towards attributing false beliefs when knowledge states are cued using a non-factive verb (``John thinks...'') than when cued indirectly (``John looks in the...''). Unlike the primary effect of knowledge states, where human sensitivity exceeds that of LMs, the magnitude of the human knowledge cue effect falls squarely within the distribution of LM effect sizes-suggesting that distributional statistics of language can in principle account for the latter but not the former in humans. These results demonstrate the value of using larger samples of open-weight LMs to test theories of human cognition and evaluate LM capacities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑