arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.17659cs.CVcs.RO
收藏

当视觉超越语言:评估和缓解VLAs中的反事实失败

When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs

University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校)

Yu Fang, Yuchun Feng, Dong Jing, Jiaqi Liu, Yue Yang, Zhenyu Wei, Daniel Szafir, Mingyu Ding

AI总结:

本文提出CAG方法,通过双分支推理方案缓解VLAs中的反事实失败,提升语言跟随和任务成功率。

摘要:

视觉-语言-动作模型(VLAs)承诺将语言指令接地于机器人控制,但实践中往往无法忠实遵循语言。当遇到缺乏强场景特定监督的指令时,VLAs会受到反事实失败的影响:它们基于数据集偏见诱导的视觉捷径行事,反复执行已学习良好的行为,并选择频繁出现在训练期间的对象,而不管语言意图。为了系统地研究这一问题,我们引入了LIBERO-CF,这是首个针对VLAs的反事实基准,通过在视觉上合理的LIBERO布局下分配替代指令来评估语言跟随能力。我们的评估揭示了反事实失败在最先进的VLAs中普遍存在但尚未充分研究。我们提出了反事实动作指导(CAG),一种简单而有效的双分支推理方案,它明确规范了VLAs中的语言条件。CAG结合了标准VLA策略和一个语言无条件的视觉-动作(VA)模块,使在动作选择期间能够进行反事实比较。这种设计减少了对视觉捷径的依赖,提高了在未观察任务上的鲁棒性,并不需要额外的演示或对现有架构或预训练模型的修改。广泛的实验展示了其在各种VLAs中的即插即用集成和一致的改进。例如,在LIBERO-CF上,CAG在语言跟随准确性上提高了9.7%,在未观察任务上的任务成功率提高了3.6%,使用无训练策略时,当与VA模型结合时,分别进一步提高了15.5%和8.5%。在现实世界评估中,CAG将反事实失败减少了9.4%,并在平均上提高了17.2%的任务成功率。

英文摘要:

Vision-Language-Action models (VLAs) promise to ground language instructions in robot control, yet in practice often fail to follow language faithfully. When presented with instructions that lack strong scene-specific supervision, VLAs suffer from counterfactual failures: they act based on vision shortcuts induced by dataset biases, repeatedly executing well-learned behaviors and selecting objects frequently seen during training regardless of language intent. To support systematic study, we introduce LIBERO-CF, the first counterfactual benchmark for VLAs that evaluates language following capability by assigning alternative, feasible task instructions under the same LIBERO scene layouts. Our evaluation reveals that counterfactual failures are prevalent yet underexplored across state-of-the-art VLAs. To mitigate these failures, we propose Counterfactual Action Guidance (CAG), a simple yet effective dual-branch inference scheme that explicitly strengthens language conditioning in VLAs. CAG combines a standard VLA policy with a parallel language-unconditioned branch, amplifying the difference between their action predictions induced by language instructions. This design reduces reliance on vision shortcuts, improves robustness on counterfactual tasks, and requires neither additional demonstrations nor modifications to the base VLA architecture or its pretrained weights. Extensive experiments demonstrate consistent improvements across diverse VLAs. On LIBERO-CF, our training-free variant, TF-CAG, improves $π_{0.5}$ by 9.7% in language grounding and 3.6% in task success. When using a separate Vision-Action (VA) module, VA-CAG achieves larger gains of 15.5% and 8.5%, respectively. In real-world evaluations, VA-CAG improves language grounding by 9.4% and task success by 17.2% on average.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑