arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.19210cs.AI
收藏

观察者,而非玩家:通过游戏观察模拟大语言模型中的理论心理论

Observer, Not Player: Simulating Theory of Mind in LLMs through Game Observation

Department of Management Information Systems, National ChengChi University(管理信息系,国立中正大学)

Jerry Wang, Ting Yiu Liu

AI总结:

通过游戏观察模拟大语言模型中的理论心理论,评估其在顺序行为中的推理能力。

摘要:

我们提出一个交互框架,用于评估大型语言模型(LLMs)是否在简单的战略环境中表现出真正的'理解'。以Rock-Paper-Scissors(RPS)为例,尽管其表面简单,但需要顺序推理、适应和策略识别。我们的系统将LLM定位为观察者,其任务是识别正在使用的策略,并阐述判断的理由。目的不是测试对Rock-Paper-Scissors本身的知识,而是探测模型是否能表现出对顺序行为的类似思维推理。为了支持系统的评估,我们提供了一个基准,包含静态策略和由良好提示规则指定的轻量动态策略。我们通过交叉熵、布里尔分数和预期值(EV)差异三种互补信号量化观察者预测与实际策略对诱导的真实分布之间的对齐程度。这些指标进一步整合到一个统一的分数中,即联合损失,平衡了校准、灵敏度和收益对齐。与策略识别率(SIR)指标一起,我们的框架不仅捕捉了预测准确性,还捕捉了模型是否能稳定识别正在发挥作用的隐含策略。演示强调了互动性、透明性和可重复性。用户可以实时调整LLM分布,可视化损失如何演变,并直接检查推理片段以确定失败发生在哪里和为什么。通过这样做,我们的系统提供了一个实用且可解释的代理,用于模拟顺序游戏中的类似思维推理,为当前LLM推理的优势和局限性提供了见解。

英文摘要:

We present an interactive framework for evaluating whether large language models (LLMs) exhibit genuine "understanding" in a simple yet strategic environment. As a running example, we focus on Rock-Paper-Scissors (RPS), which, despite its apparent simplicity, requires sequential reasoning, adaptation, and strategy recognition. Our system positions the LLM as an Observer whose task is to identify which strategies are being played and to articulate the reasoning behind this judgment. The purpose is not to test knowledge of Rock-Paper-Scissors itself, but to probe whether the model can exhibit mind-like reasoning about sequential behavior. To support systematic evaluation, we provide a benchmark consisting of both static strategies and lightweight dynamic strategies specified by well-prompted rules. We quantify alignment between the Observer's predictions and the ground-truth distributions induced by actual strategy pairs using three complementary signals: Cross-Entropy, Brier score, and Expected Value (EV) discrepancy. These metrics are further integrated into a unified score, the Union Loss, which balances calibration, sensitivity, and payoff alignment. Together with a Strategy Identification Rate (SIR) metric, our framework captures not only predictive accuracy but also whether the model can stably identify the latent strategies in play. The demo emphasizes interactivity, transparency, and reproducibility. Users can adjust LLM distributions in real time, visualize losses as they evolve, and directly inspect reasoning snippets to identify where and why failures occur. In doing so, our system provides a practical and interpretable proxy for mind-like inference in sequential games, offering insights into both the strengths and limitations of current LLM reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑