Word Embeddings Inherently Recover the Conceptual Organization of the Human Mind

FANToM：用于在交互中对机器心智理论进行压力测试的基准

FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactions

本文提出 FANToM 基准，通过信息不对称对话中的问答来压力测试大语言模型的心智理论能力，并发现现有模型即使经过思维链推理或微调仍显著落后于人类。

Unpacking Large Language Models with Conceptual Consistency

If a Large Language Model (LLM) answers "yes" to the question "Are mountains tall?" then does it know what a mountain is? Can you rely on it responding correctly or incorrectly to other questions about mountains? The success of Large Language Models (LLMs) indicates they are increasingly able to answer queries like these accurately, but that ability does not necessarily imply a general understanding of concepts relevant to the anchor query. We propose conceptual consistency to measure a LLM's understanding of relevant concepts. This novel metric measures how well a model can be characterized b…

在贝叶斯心智理论中为关于信念的语言提供基础

Grounding Language about Belief in a Bayesian Theory-of-Mind

该研究将信念陈述语义建立在贝叶斯心智理论基础上，构建能联合推断智能体目标、信念与计划的框架，经门与钥匙网格世界谜题实验验证，其对人类信念归因的拟合度优于多种基线方法。

Word meaning in minds and machines

Machines have achieved a broad and growing set of linguistic competencies, thanks to recent progress in Natural Language Processing (NLP). Psychologists have shown increasing interest in such models, comparing their output to psychological judgments such as similarity, association, priming, and comprehension, raising the question of whether the models could serve as psychological theories. In this article, we compare how humans and machines represent the meaning of words. We argue that contemporary NLP systems are fairly successful models of human word similarity, but they fall short in many o…

概念结构在人类认知中具有一致性，但在大语言模型中并非如此

Conceptual structure coheres in human cognition but not in large language models

本研究通过认知心理学方法比较人类与大语言模型的概念结构，发现人类概念结构跨文化、语言和方法稳健一致，而大语言模型的概念结构因任务而异，内部一致性较低，揭示了当代机器语言的局限性。

一种用于文本概念多样性的新度量指标提案

A proposed new metric for the conceptual diversity of a text

本文提出一种标准化度量指标，用于量化文本的概念多样性，通过示例验证其有效性，为自然语言处理与语义研究提供新工具。