AsgardBench -- 评估基于视觉的交互式规划在最小反馈下的表现
AsgardBench -- Evaluating Visually Grounded Interactive Planning Under Minimal Feedback
AI总结：
AsgardBench旨在评估基于视觉的高阶动作序列生成和交互式规划，重点在于执行过程中基于视觉观察而非导航或低级操作的计划适应。通过限制代理输入为图像、动作历史和轻量级成功/失败信号，该基准测试强调条件分支和计划修复。
摘要：
AsgardBench旨在评估基于视觉的高阶动作序列生成和交互式规划，重点在于执行过程中基于视觉观察而非导航或低级操作的计划适应。在具身AI基准测试领域，AsgardBench针对交互式规划能力类别，该类别比离线高阶规划更复杂，因为它要求代理根据环境反馈修改计划，但又不同于低阶执行。与以往将推理与导航混为一谈或提供丰富纠正反馈替代感知的具身AI基准测试不同，AsgardBench限制代理输入为图像、动作历史和轻量级成功/失败信号，在受控模拟器中隔离交互式规划，不包含低阶控制噪声。该基准测试包含108个任务实例，涵盖12种任务类型，每个任务通过物体状态、放置和场景配置系统性变化。这些受控变化创建了条件分支，在其中单条指令可能需要不同的动作序列，取决于代理观察到的内容，强调执行过程中的条件分支和计划修复。对领先视觉语言模型的评估显示，没有视觉输入时性能急剧下降，揭示了视觉 grounding 和状态跟踪的弱点，最终影响交互式规划。该基准测试聚焦于更狭窄的问题：模型是否能实际利用所见内容在事情不如预期时调整计划？
英文摘要：
With AsgardBench we aim to evaluate visually grounded, high-level action sequence generation and interactive planning, focusing specifically on plan adaptation during execution based on visual observations rather than navigation or low-level manipulation. In the landscape of embodied AI benchmarks, AsgardBench targets the capability category of interactive planning, which is more sophisticated than offline high-level planning as it requires agents to revise plans in response to environmental feedback, yet remains distinct from low-level execution. Unlike prior embodied AI benchmarks that conflate reasoning with navigation or provide rich corrective feedback that substitutes for perception, AsgardBench restricts agent input to images, action history, and lightweight success/failure signals, isolating interactive planning in a controlled simulator without low-level control noise. The benchmark contains 108 task instances spanning 12 task types, each systematically varied through object state, placement, and scene configuration. These controlled variations create conditional branches in which a single instruction can require different action sequences depending on what the agent observes, emphasizing conditional branching and plan repair during execution. Our evaluations of leading vision language models show that performance drops sharply without visual input, revealing weaknesses in visual grounding and state tracking that ultimately undermine interactive planning. Our benchmark zeroes in on a narrower question: can a model actually use what it sees to adapt a plan when things do not go as expected?
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
