LLM时代：战争迷雾下大型语言模型的推理、外交与可靠性的战略1v1基准测试
Age of LLM: A Strategic 1v1 Benchmark for Reasoning, Diplomacy and Reliability of Large Language Models under Fog of War
AI总结：
提出Age of LLM基准，在13x7网格上让两个LLM进行1v1对战，包含战争迷雾、完全外交和严格JSON格式可靠性维度，通过54场比赛评估15个推理模型，发现核冲刺主导、外交频繁但极少达成、非法动作反映信念追踪，并初步关联可靠性与获胜。
摘要：
我们引入了Age of LLM，一个回合制1v1基准测试，其中两个LLM在13x7网格上对战，目标是摧毁敌方基地。三个刻意设计的压力因素：战争迷雾、完全外交（消息、停火、最后通牒；铀矿保密）以及可靠性维度——每回合必须遵循严格的JSON模式，非法动作被静默丢弃。引擎是私有的，每场比赛使用随机地图种子和对手，减轻了影响公开基准的数据污染。模型接收（几乎）仅包含规则的提示，没有建造顺序建议（数据收集期间存在两个战术种子短语；见第2.7节）。我们对15个推理模型进行了54场比赛和5,258个动作的基准测试。发现：（1）核冲刺占主导（在规则一致的v0.11+子语料库中占78%；全语料库中占85%），具有单发射器特征，在秘密同时发射规则下主要是机械性的，而非认知威慑失败；（2）军事征服罕见但更快（12.3回合对比18.9回合）；（3）外交频繁但几乎从未达成；（4）约58%的非法动作是战争迷雾/状态错误，使非法动作率成为信念追踪的度量；（5）——最不成熟且我们标记为探索性的——一个弱关联将可靠性与获胜联系起来。语料库小、不平衡且未交换阵营，因此排名是初步的描述性视图，而非贡献。除了排名，逐回合的动作和消息轨迹使语料库成为观察LLM在对抗性不确定性下推理的窗口——它们的信念追踪、自发欺骗以及每个模型的认知“人格”——我们将其定位为未来研究方向。我们发布了回放格式、等距查看器和所有回放；引擎源代码可按需提供。
英文摘要：
We introduce Age of LLM, a turn-based 1v1 benchmark in which two LLMs face off on a 13x7 grid to destroy the enemy base. Three stressors are deliberate: fog of war, full diplomacy (messages, ceasefires, ultimatums; uranium kept secret), and a reliability dimension where every turn must follow a strict JSON schema and an illegal action is silently discarded. The engine is private and each match uses a fresh random map seed and opponent, mitigating the data contamination that affects public benchmarks. Models receive a (near) rule-only prompt with no build-order advice (two tactical seed phrases were present during data collection; see Section 2.7). We benchmark 15 reasoning models across 54 matches and 5,258 actions. Findings: (1) the nuclear rush dominates (78% on the rules-coherent v0.11+ sub-corpus; 85% corpus-wide) with a sole-launcher signature that is largely mechanical under secret-simultaneous launch rules, not a cognitive deterrence failure; (2) military conquest is rare but faster (12.3 vs 18.9 turns); (3) diplomacy is prolific yet almost never consummated; (4) ~58% of illegal actions are fog/state errors, making the illegal-action rate a measure of belief-tracking; (5) -- the least established, and the only one we label exploratory -- a weak link associates reliability with winning. The corpus is small, unbalanced and not side-swapped, so the ranking is a preliminary descriptive view, not a contribution. Beyond ranking, the turn-by-turn traces of actions and messages make the corpus a lens on how LLMs reason under adversarial uncertainty -- their belief-tracking, spontaneous deception, and per-model cognitive "personas" -- which we frame as a future research direction. We release the replay format, an isometric viewer and all replays; engine source on request.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
