从黑盒到透明心智：多模态大语言模型中心智理论的评估与增强
From Black Boxes to Transparent Minds: Evaluating and Enhancing the Theory of Mind in Multimodal Large Language Models
AI总结：
本研究针对现有机器心智理论评估多聚焦单模态黑盒模型的问题，构建多模态ToM测试数据集GridToM，从注意力头内部机制验证多模态大模型的ToM能力，并提出轻量免训练方法增强该能力。
摘要：
随着大语言模型的发展，人们越来越期望它们能模拟类人类的心智理论（Theory of Mind, ToM）以协助完成日常任务。然而，现有的机器心智理论评估方法主要聚焦于单模态模型，且大多将这些模型视为黑盒，缺乏对其内部机制的可解释性探索。针对这一问题，本研究采用基于内部机制的方法，对多模态大语言模型（Multimodal Large Language Models, MLLMs）的心智理论开展可解释性驱动的评估。具体而言，我们首先构建了一个多模态心智理论测试数据集GridToM，该数据集包含多样化的信念测试任务和多视角感知信息。随后，我们的分析表明，多模态大模型中的注意力头能够区分不同视角的认知信息，为模型具备心智理论能力提供了证据。此外，我们提出了一种轻量、免训练的方法，通过沿注意力头方向进行调整，显著增强了模型所展现出的心智理论能力。
英文摘要：
As large language models evolve, there is growing anticipation that they will emulate human-like Theory of Mind (ToM) to assist with routine tasks. However, existing methods for evaluating machine ToM focus primarily on unimodal models and largely treat these models as black boxes, lacking an interpretative exploration of their internal mechanisms. In response, this study adopts an approach based on internal mechanisms to provide an interpretability-driven assessment of ToM in multimodal large language models (MLLMs). Specifically, we first construct a multimodal ToM test dataset, GridToM, which incorporates diverse belief testing tasks and perceptual information from multiple perspectives. Next, our analysis shows that attention heads in multimodal large models can distinguish cognitive information across perspectives, providing evidence of ToM capabilities. Furthermore, we present a lightweight, training-free approach that significantly enhances the model's exhibited ToM by adjusting in the direction of the attention head.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
