arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.24697cs.AI
收藏

当前智能体能否缩小发现到应用的差距?Minecraft中的一个案例研究

Can Current Agents Close the Discovery-to-Application Gap? A Case Study in Minecraft

University of California, Los Angeles(加州大学洛杉矶分校);Peking University(北京大学);Amazon(亚马逊)

Zhou Ziheng, Huacong Tang, Jinyuan Zhang, Haowei Lin, Bangcheng Yang, Qian Long, Fang Sun, Yizhou Sun, Yitao Liang, Ying Nian Wu, Demetri Terzopoulos, Xiaofeng Gao

AI总结:

本文通过Minecraft中的SciCrafter基准测试,探讨了智能体在发现因果规律并将其应用于构建功能系统(发现-应用循环)方面的能力,发现前沿模型在该任务中的成功率约为26%,揭示了知识识别和问题提出能力成为当前AI的瓶颈。

摘要:

发现因果规律并将其应用于构建功能性系统——发现-应用循环——是通用智能的标志,但评估这一能力受到科学发现与现实世界工程之间巨大复杂性差距的阻碍。我们引入了基于Minecraft的SciCrafter基准测试,通过参数化的红石电路任务来操作化这一循环。智能体必须按照指定的模式(例如同时或按时间序列)点燃灯泡;扩大目标参数会显著增加构建复杂性和所需知识,迫使真正的发现而非依赖记忆中的解决方案。在通用目的代码智能体框架下评估前沿模型,包括GPT-5.2、Gemini-3-Pro和Claude-Opus-4.5,我们发现所有模型均在约26%的成功率处停滞。为了诊断这些失败,我们将循环分解为四个能力——知识差距识别、实验发现、知识整合和知识应用,并设计了针对性的干预措施,其边际贡献作为相应差距的代理。我们的分析表明,尽管通用知识应用能力仍然是所有模型中最大的差距,但对前沿模型而言,知识差距识别开始成为主要障碍——表明瓶颈正从解决正确的问题转变为提出正确的问题。我们发布了SciCrafter作为未来研究AI系统在完整发现-应用循环中导航的诊断探针。

英文摘要:

Discovering causal regularities and applying them to build functional systems--the discovery-to-application loop--is a hallmark of general intelligence, yet evaluating this capacity has been hindered by the vast complexity gap between scientific discovery and real-world engineering. We introduce SciCrafter, a Minecraft-based benchmark that operationalizes this loop through parameterized redstone circuit tasks. Agents must ignite lamps in specified patterns (e.g., simultaneously or in timed sequences); scaling target parameters substantially increases construction complexity and required knowledge, forcing genuine discovery rather than reliance on memorized solutions. Evaluating frontier models including GPT-5.2, Gemini-3-Pro, and Claude-Opus-4.5 under a general-purpose code agent scaffold, we find that all plateau at approximately 26% success rate. To diagnose these failures, we decompose the loop into four capacities--knowledge gap identification, experimental discovery, knowledge consolidation, and knowledge application--and design targeted interventions whose marginal contributions serve as proxies for corresponding gaps. Our analysis reveals that although the general knowledge application capability still remains as the biggest gap across all models, for frontier models the knowledge gap identification starts to become a major hurdle--indicating the bottleneck is shifting from solving problems right to raising the right problems for current AI. We release SciCrafter as a diagnostic probe for future research on AI systems that navigate the full discovery-to-application loop.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑