随着ChatGPT及其他同等规模的SotA大语言模型（LLM）取得成功，科学文献中出现了这类模型涌现出类人社会推理能力、尤其是心智理论（Theory of Mind, ToM）的论断。一方面，研究者采用与心理学实验类似的任务范式，已成功验证了这些ToM能力的存在（Kosinski, 2023）；另一方面，后续研究表明，当任务被稍作改动时，这些能力就会消失（Ullman, 2023）。本研究构建了一个包含68个任务的新型数据集，用于探究大语言模型的ToM能力，其中涵盖了可能具有挑战性的任务变体，这些变体被划分为10个复杂度等级，借此为揭示大语言模型在应对这类任务变体时面临的挑战提供新的见解。我们在本研究的数据集与Kosinski（2023）提出的数据集上，评估了四款SotA开源大语言模型的ToM表现。所有受评估模型的整体目标准确率都较低，表明其仅具备有限程度的ToM能力。在两个数据集的简单复杂度等级任务上，这些大语言模型的表现相近；但我们发现，所有受测大语言模型在需要意识到智能体知晓其环境中自动状态变化的任务上表现普遍较差，即便这些变化已被明确告知模型也是如此。对于通过替换介词改变对象间关系的任务复杂化设置，我们注意到所有模型的性能均出现下降，其中对混合专家模型（mixture-of-experts model）的影响最为显著。我们的按复杂度分组的任务数据集，为后续研究如何稳定并提升大语言模型的ToM能力提供了方向。
英文摘要：
With the success of ChatGPT and other similarly sized SotA LLMs, claims of emergent human like social reasoning capabilities, especially Theory of Mind (ToM), in these models have appeared in the scientific literature. On the one hand those ToM-capabilities have been successfully tested using tasks styled similar to those used in psychology (Kosinski, 2023). On the other hand, follow up studies showed that those capabilities vanished when the tasks were slightly altered (Ullman, 2023). In this work we introduce a novel dataset of 68 tasks for probing ToM in LLMs, including potentially challenging variations which are assigned to 10 complexity classes. This way it is providing novel insights into the challenges LLMs face with those task variations. We evaluate the ToM performance of four SotA open source LLMs on our dataset and the dataset introduced by (Kosinski, 2023). The overall low goal accuracy across all evaluated models indicates only a limited degree of ToM capabilities. The LLMs' performance on simple complexity class tasks from both datasets are similar. Whereas we find a consistent tendency in all tested LLMs to perform poorly on tasks that require the realization that an agent has knowledge of automatic state changes in its environment, even when those are spelled out to the model. For task complications that change the relationship between objects by replacing prepositions, we notice a performance drop in all models, with the strongest impact on the mixture-of-experts model. With our dataset of tasks grouped by complexity we offer directions for further research on how to stabilize and advance ToM capabilities in LLM.
Mind Your Theory: Theory of Mind Goes Deeper Than Reasoning
Hebrew University of Jerusalem（海法大学）；MPI for Biological Cybernetics（生物感知研究所）；Centre for AI and Machine Learning, AU（人工智能与机器学习中心）；IoPPN, King’s College London（国王学院伦敦人工智能与感知神经科学中心）
Probing the effectiveness of World Models for Spatial Reasoning through Test-time Scaling
MILA – Quebec AI Institute（魁北克人工智能研究所）；Polytechnique Montréal（蒙特利尔理工学院）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；Institute for Machine Learning, Johannes Kepler University Linz（林茨约瑟夫·夫兰克大学机器学习研究所）；Nankai University（南开大学）
Construction Grammar Provides Unique Insight into Neural Language Models
Center for Information and Language Processing, LMU Munich（慕尼黑大学信息与语言处理中心）；Munich Center of Machine Learning（慕尼黑机器学习中心）；Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）
Understanding Artificial Theory of Mind: Perturbed Tasks and Reasoning in Large Language Models
Bonn-Aachen International Center for Information Technology (b-it)（波恩-埃森国际信息科技中心）；University of Bonn（波恩大学）；Lamarr Institute for Machine Learning and Artificial Intelligence（拉马尔机器学习与人工智能研究所）；Research Center Trustworthy Data Science and Security (RC-Trust)（可信数据科学与安全研究中心）；University of Duisburg-Essen（埃森-杜伊斯堡大学）
本研究通过扰动任务和链式推理提示探讨大语言模型的理论 of mind能力，发现其鲁棒性受扰动影响显著，CoT提示虽提升整体表现，但对某些扰动类别却降低准确性。
Probing the Robustness of Large Language Models Safety to Latent Perturbations
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际 Graduate School, 清华大学）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Fudan University（复旦大学）；The University of Hong Kong（香港大学）
Do Large Language Models Possess a Theory of Mind? A Comparative Evaluation Using the Strange Stories Paradigm
ELTE Research Centre for Linguistics（ELTE语言学研究中心）；Department of Cognitive Science, Faculty of Natural Sciences, Budapest University of Technology and Economics（布达佩斯技术与经济大学自然科学学院认知科学系）；Institute of Mathematics, Eötvös Lóránd University（欧多布雷尼大学数学研究所）
MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Models
Shenyang Institute of Automation, Chinese Academy of Sciences（中国科学院沈阳自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）；MMLab, CUHK（CUHK多模态实验室）
MIST: Towards Multi-dimensional Implicit BiaS Evaluation of LLMs for Theory of Mind
School of Software（软件学院）；Shandong University（山东大学）；National University of Singapore（新加坡国立大学）；School of Psychology（心理学学院）；Hainan Normal University（海南师范大学）
Department of Statistics, University of Chicago（芝加哥大学统计系）；Department of Statistics and Department of Computer Science, University of Chicago（芝加哥大学统计系和计算机科学系）
Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-trained Language Models
University of Southern California（南加州大学）
最近的工作表明，预训练语言模型（PTLM），如BERT，拥有一定的常识和事实知识。他们认为，通过预测屏蔽词，将PTLM用作“神经知识库”是有希望的。令人惊讶的是，我们发现这可能不适用于数字常识知识（例如，鸟通常有两条腿。在本文中，我们调查是否以及在何种程度上，我们可以从PTLM以及这个过程的鲁棒性的数值常识知识。为了研究这一点，我们引入了一个新的探测任务，诊断数据集NumerSense包含13.6k个屏蔽词预测探测器（10.5k用于微调，3.1k用于测试）。我们的分析表明：（1）在任何微调之前，BERT及其更强的变体RoBERTa在诊断数据集上表现不佳;（2）远程监督的微调带来了一些改进;（3）与人类表现相比，最好的监督模型仍然表现不佳（准确率为54.06% vs 96.3%）。
Intercepting the Kangaroo: Experimental Astrolinguistics with Constructed Lexicons, Active Probing, and Large Language Models as Informants and Hypothesis Proposers
Mind the Gap: A Review of Arabic Post-Training Datasets and Their Limitations
Refine AI；ASAS AI；University of Bisha（比沙大学）；University College London（伦敦大学学院）；King Salman Global Academy for Arabic（萨勒曼全球阿拉伯学院）；University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；King Abdulaziz University（阿卜杜勒阿齐兹大学）；HUMAIN
本文综述 Hugging Face Hub 上的阿拉伯语后训练数据集，从能力、可引导性、对齐和鲁棒性等维度评估其质量与局限，并提出未来建设建议。
Speak Your Mind: The Speech Continuation Task as a Probe of Voice-Based Model Bias
Department of Speech, Music and Hearing, KTH Royal Institute of Technology（语音、音乐与听觉系，皇家理工学院）；Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-lab（格勒诺布尔阿尔卑斯大学，法国国家科学研究中心，格勒诺布尔理工学院，GIPSA实验室）
Turning Logic Against Itself : Probing Model Defenses Through Contrastive Questions
Ubiquitous Knowledge Processing Lab (UKP Lab), Department of Computer Science and Hessian Center for AI (hessian.AI), Technical University of Darmstadt（德累斯顿技术大学计算机科学系、海斯堡人工智能中心（hessian.AI）、通用知识处理实验室（UKP Lab））
针对推理驱动的隐蔽越狱漏洞，本文提出 POATE 对比推理攻击，并设计 Intent-Aware CoT 与 Reverse Thinking CoT 以检测恶意意图、拒绝有害输出。
Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing
State Key Laboratory of Multimodal Artificial Intelligence System, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Department of Computer Science, The University of Manchester（曼彻斯特大学计算机科学系）；Department of Language Science and Technology, Hong Kong Polytechnic University（香港理工大学语言科学与技术系）
Mind Scramble: Unveiling Large Language Model Psychology Via Typoglycemia
University of Science and Technology of China (USTC)（中国科学技术大学）；Squirrel AI（松鼠AI）；National University of Singapore (NUS)（新加坡国立大学）；NanYang Technological University (NTU)（南洋理工大学）；The Hong Kong University of Science and Technology (HKUST)（香港科技大学）
Answer Probing-Guided Search for Diverse Solution Exploration of LLMs
University of Science and Technology of China（中国科学技术大学）；Zhongguancun Academy（中关村学院）；Alibaba Group（阿里巴巴集团）；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）
