GPTNT：在《保持通话，无人爆炸》中基准测试多模态智能体之间的实时协作
GPTNT: Benchmarking Real-Time Collaboration Between Multimodal Agents on Keep Talking And Nobody Explodes
AI总结：
提出GPTNT基准，基于合作游戏《保持通话，无人爆炸》测试多模态智能体在实时、信息不对称条件下的协作能力，发现当前最先进系统无法在实时中拆除任何炸弹。
摘要：
多模态模型越来越多地被部署用于与人类或其他人工智能体协作解决问题。现有基准测试表明，这些模型具备许多所需的组件能力，但协作中的条件，包括时间压力、信息不对称和不完美沟通，通常被孤立研究。我们引入GPTNT，一个基于合作视频游戏《保持通话，无人爆炸》构建的基准测试，其中两个智能体必须协调配合，在实时倒计时中拆除程序生成的炸弹谜题。一个智能体可以看到并操作炸弹，但没有拆解说明；另一个智能体有说明，但看不到也无法操作炸弹。两个智能体都无法单独成功：成功需要有效且高效的沟通。与回合制代理不同，GPTNT要求智能体异步行动并实时通信。GPTNT旨在将协作与依赖记忆解决方案分开：可以隐藏说明手册、合作伙伴或两者，以隔离模型从已有知识中即时推导出的内容。我们表明，GPTNT对最先进的系统构成了重大挑战：我们测试的闭源或开源模型没有一个能在实时中拆除单个炸弹，而人类玩家可以轻松做到。通过控制实验，我们识别出状态跟踪、时间压力下的高效行动、歧义处理和错误恢复方面的关键弱点。我们发布GPTNT作为协作性能的基准测试，当前评估未对此进行测量。由于它在真实游戏上运行，GPTNT受益于程序生成并继承了活跃的模组社区，使得基准测试能够随着模型改进而演变，而不是被解决一次后就废弃。
英文摘要：
Multimodal models are increasingly deployed to solve tasks collaboratively with humans or other artificial agents. While existing benchmarks show that they possess the fundamental capabilities, the various conditions that coincide when collaborating---time pressure, information asymmetry, and imperfect communication---have traditionally been studied in isolation. To address this gap, we introduce GPTNT, a benchmark built on the cooperative video game Keep Talking and Nobody Explodes, in which two agents must coordinate to defuse procedurally generated bomb puzzles against a live countdown. One agent has access to the bomb but not the instructions for defusing it; the other holds the instructions but cannot see or manipulate the bomb. Neither agent can succeed alone: the task requires contributions from both, and is solvable only through effective, efficient communication. We remove turn-taking proxies or simplifications, instead requiring agents to act asynchronously and communicate in real time. GPTNT is designed to expose how models collaborate versus how they perform alone: the instruction manual, the partner, or both, can optionally be withheld to surface what a model has memorised versus what it derives in the moment. We demonstrate that GPTNT poses a considerable challenge to the state-of-the-art: not one of the closed- and open-source models we test defuses a single bomb in real time, a bar that human players clear. In a range of controlled experiments, we explore where capabilities break down, identifying critical weaknesses in state tracking, efficient acting within the time budget, handling ambiguity, and error recovery. Since it runs on the real game, GPTNT benefits from procedural generation and inherits a living modding community: as models improve, the benchmark can be evolved to remain challenging, rather than being solved once and retired.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
