GameReplica：面向视觉语言代理的黑盒视觉游戏复现基准
GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agents
另 2 家 · 查看机构详情
- Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）
- School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
- Zhongguancun Laboratory（中关村实验室）
- Xiaomi EV（小米汽车）
- Alibaba DAMO Academy（阿里巴巴达摩院）
机构由 AI 辅助整理，请以论文原文为准。
浏览论文内容
中文总结 AI 辅助
GameReplica提出黑盒游戏复现基准，通过闭环框架评估视觉语言代理仅凭截图和交互归纳规则并生成可验证游戏的能力，实验显示当前代理视觉复现强于机制复现。
中文摘要 AI 辅助
编码代理基准通常在目标行为已通过文本、代码或演示指定后评估实现。现有研究已广泛评估编码代理根据文本规范生成程序的能力。然而，在黑盒条件下，既无源代码也无文档可用时，代理能否仅通过视觉观察和主动交互归纳规则，并将目标系统复现为可验证的可执行系统，仍未被充分探索。为此，我们提出GameReplica，一个用于端到端黑盒游戏复现的闭环评估框架，覆盖感知、探索、归纳、复现和验证的完整流程。GameReplica包含125个任务，涵盖5个核心机制家族的25个游戏，每个游戏在五个难度级别上实例化。任务要求代理仅通过截图和动作接口访问目标游戏，从像素反馈和交互结果中归纳关键视觉元素和游戏规则，并生成一个自包含、可运行的游戏复现品，该复现品可由外部程序自动验证。实验表明，当前编码代理在端到端黑盒复现中仍面临重大挑战：最佳模型（Claude Opus 4.8）达到71.6%的总分，而其余模型仅得分4.0%至42.9%。进一步分析揭示所有模型的一致模式：视觉保真度得分显著高于实现和规则一致性得分，表明代理更容易复现视觉外观而非游戏机制。难度级别进一步放大性能差距：从L1到L5，较弱代理的总分急剧下降，而最佳代理的总分仅略有下降。
英文摘要
Coding-agent benchmarks usually evaluate implementation after the target behavior has been specified in text, code, or demonstrations. Existing research has extensively evaluated the ability of coding agents to generate programs from textual specifications. However, under black-box conditions where neither source code nor documentation is available, it remains underexplored whether an agent can induce the rules solely through visual observation and active interaction and reproduce the target system as a verifiable executable system. To this end, we present GameReplica, a closed-loop evaluation framework for end-to-end black-box game replication that covers the full perception, exploration, induction, reproduction, and verification pipeline. GameReplica comprises 125 tasks spanning 25 games across 5 core mechanism families, with each game instantiated at five difficulty levels. The tasks require an agent to access the target game only through screenshots and an action interface, induce the key visual elements and gameplay rules from pixel feedback and interaction outcomes, and generate a self-contained, runnable game replica that can be automatically verified by an external program. Experiments show that current coding agents still face substantial challenges in end-to-end black-box replication: the best-performing model (Claude Opus 4.8) achieves an overall score of 71.6\%, while the remaining models score only 4.0\%--42.9\%. Further analysis reveals a consistent pattern across all models: visual-fidelity scores are substantially higher than implementation- and rule-consistency scores, indicating that agents replicate visual appearance more readily than game mechanics. The difficulty levels further amplify the performance gap: from L1 to L5, the overall score of weaker agents drops sharply, whereas that of the best-performing agent declines only slightly.
