SpireBench：使用《杀戮尖塔》评估智能体模型的长程战略博弈能力
SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire
AI总结：
SpireBench是首个提供完整《杀戮尖塔》游戏的长程战略基准，测试20个LLM发现其表现接近普通人类但远逊于专家，可定位智能体长程战略推理的失败原因。
摘要：
智能体大语言模型如今可处理复杂任务，但长程战略推理的研究仍不充分：多数智能体基准测试是短程且分幕式的，长任务上的表现难以拆解为存储知识、推理能力和运气的贡献。我们推出SpireBench，让模型游玩完整的商业卡组构建游戏《杀戮尖塔》（Slay the Spire，StS），其中早期决策会决定长期成败。在游戏最高难度下对20个大语言模型（LLM）的测试显示，前沿模型击败了启发式基线，表现与普通人类玩家相当，但仍远逊于专家玩家。随着游戏进程延长，这一差距会扩大：最强模型在第一幕的胜率与专家相当，之后便会落后。区分强弱模型的关键在于它们每场战斗的表现；所有模型在整个游戏过程中都存在风险误判，倾向于保留生命值而非消耗生命值来提升实力。我们对所有游戏术语进行了全面且自洽的重命名，结果表明模型的部分表现依赖于对该游戏的先验知识，且模型无法通过跨游戏回合写入的记忆提升表现，这些记忆往往是错误且过于谨慎的。我们的测试框架是首个为模型提供完整原始游戏的框架，我们将其与现有测试框架进行了对比。总体而言，SpireBench是一个未达饱和状态的长程基准测试，配备了可定位智能体失败的位置和原因的工具。
英文摘要：
Agentic large language models now tackle complex tasks, yet strategic reasoning over long horizons is under-studied: most agentic benchmarks are short and episodic, and performance on long tasks is hard to dissect into stored knowledge, reasoning and luck. We introduce SpireBench, in which models play the complete commercial deck-builder Slay the Spire (StS), where early decisions decide long-term success. Across 20 large language models (LLMs) at the game's highest difficulty, frontier models beat heuristic baselines and match the average human player but remain far behind experts. This gap widens as the run lengthens: the strongest models win the first act as often as experts, then fall behind. What separates strong models from weak is how well they play each fight; all of them misjudge risk over a run, preserving health rather than spending it to grow stronger. A comprehensive, self-consistent renaming of every game term shows that part of their performance rests on prior knowledge of the game, and models do not improve from the memories they write across runs, which are often wrong and overly cautious. Our harness is the first to give a model the complete original game, and we compare it with existing harnesses. Overall, SpireBench offers an unsaturated long-horizon benchmark with tools to locate where and why agents fail.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
