MindGames：利用动态认知模态逻辑针对大语言模型的心理理论开展研究
MindGames: Targeting Theory of Mind in Large Language Models with Dynamic Epistemic Modal Logic
AI总结：
该研究提出MindGames框架，用动态认知逻辑分离心理理论组件并生成可控问题，结合新语言化技术测试大模型，发现部分规模扩展未带来优于随机的表现，GPT-4能力仍有提升空间，且公开了相关代码与数据集。
摘要：
心理理论（ToM）是智能的关键组成部分，但其评估方式仍存激烈争议。过往研究将人类ToM评估应用于自然语言处理模型，采用人工设计的标准化测试或基于规则的模板，但这些方法多聚焦简单推理，仍需进一步验证。本文利用动态认知逻辑分离出ToM的特定组件，并生成可控的问题，还提出了新的语言化技术，将这些问题转化为英语自然语言表述。研究结果显示，部分语言模型的规模扩展（从70M到6B、从350M到174B）并未持续带来优于随机猜测的表现。尽管GPT-4展现出更优的认知推理能力，但仍有提升空间。本文的代码和数据集已公开（https://huggingface.co/datasets/sileod/mindgames 、https://github.com/sileod/llm-theory-of-mind ）。
英文摘要：
Theory of Mind (ToM) is a critical component of intelligence but its assessment remains the subject of heated debates. Prior research applied human ToM assessments to natural language processing models using either human-created standardized tests or rule-based templates. However, these methods primarily focus on simplistic reasoning and require further validation. Here, we leverage dynamic epistemic logic to isolate a particular component of ToM and to generate controlled problems. We also introduce new verbalization techniques to express these problems in English natural language. Our findings indicate that some language model scaling (from 70M to 6B and 350M to 174B) does not consistently yield results better than random chance. While GPT-4 demonstrates superior epistemic reasoning capabilities, there is still room for improvement. Our code and datasets are publicly available (https://huggingface.co/datasets/sileod/mindgames , https://github.com/sileod/llm-theory-of-mind )
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
