GameWorld: 向标准化和可验证的多模态游戏代理评估迈进
GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents
AI总结：
GameWorld提出一个标准化且可验证的多模态游戏代理评估基准，包含34种游戏和170个任务，通过可验证的指标评估代理能力，揭示了当前代理在视频游戏中与人类能力的差距。
摘要：
为实现真实世界交互的具身通用代理，多模态大语言模型（MLLM）代理仍面临挑战性的延迟、稀疏反馈和不可逆错误。视频游戏提供了一个理想的测试平台，具有丰富的视觉观察和闭环交互，要求精细感知、长周期规划和精确控制。然而，目前系统评估这些能力受到异构动作接口和启发式验证的阻碍。为此，我们引入GameWorld，一个针对浏览器环境中的多模态游戏代理标准化和可验证评估的基准。研究了两种游戏代理接口：（i）计算机使用代理直接发射键盘和鼠标控制，（ii）通用多模态代理通过确定性语义动作解析在语义动作空间中行动。GameWorld包含34种多样游戏和170个任务，每个任务配对有状态可验证的指标用于结果评估。18组模型-接口对的结果表明，即使是最能干的代理也远未达到人类在视频游戏中的能力。广泛的重复完整基准重跑实验展示了基准的鲁棒性，而进一步研究实时交互、上下文记忆敏感性和动作有效性揭示了游戏代理面临的更多挑战。通过提供标准化、可验证和可重复的评估框架，GameWorld为推进多模态游戏代理研究奠定了坚实基础。项目页面位于https://gameworld-bench.github.io。
英文摘要：
Towards an embodied generalist for real-world interaction, Multimodal Large Language Model (MLLM) agents still suffer from challenging latency, sparse feedback, and irreversible mistakes. Video games offer an ideal testbed with rich visual observations and closed-loop interaction, demanding fine-grained perception, long-horizon planning, and precise control. However, systematically evaluating these capabilities is currently hindered by heterogeneous action interfaces and heuristic verification. To this end, we introduce GameWorld, a benchmark designed for standardized and verifiable evaluation of MLLMs as generalist game agents in browser environments. Two game agent interfaces are studied: (i) computer-use agents that directly emit keyboard and mouse controls, and (ii) generalist multimodal agents that act in a semantic action space via deterministic Semantic Action Parsing. GameWorld contains 34 diverse games and 170 tasks, each paired with state-verifiable metrics for outcome-based evaluation. The results across 18 model-interface pairs suggest that even the best performing agent is far from achieving human capabilities on video games. Extensive experiments of repeated full-benchmark reruns demonstrate the robustness of the benchmark, while further studies on real-time interaction, context-memory sensitivity, and action validity expose more challenges ahead for game agents. Together, by offering a standardized, verifiable, and reproducible evaluation framework, GameWorld lays a robust foundation for advancing research on multimodal game agents and beyond. The project page is at https://gameworld-bench.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
