通过静态-动态解耦实现高效长程视觉-语言-动作模型
Efficient Long-Horizon Vision-Language-Action Models via Static-Dynamic Disentanglement
AI总结:
提出DySta框架,通过将视觉输入解耦为多级静态和动态令牌,减少上下文长度并复用KV缓存,实现高效多帧集成和推理,在基准测试和真实任务中显著提升性能。
摘要:
视觉-语言-动作(VLA)模型最近成为通用机器人控制的一种有前景的范式。基于视觉-语言模型(VLM)架构,VLA模型根据视觉观察和语言指令预测动作,在任务中实现了强大的性能和泛化能力。然而,VLA模型面临两个主要挑战:输入帧的有限上下文窗口,以及由于二次注意力复杂性和大参数数量导致的低效推理。为此,我们提出了DySta,一个将视觉输入解耦为多级静态和动态令牌的框架,使得(1)在帧间保留静态令牌的单一副本以显著减少上下文长度,以及(2)通过轻量级重缓存门(仅在必要时更新)重用静态令牌的键值(KV)缓存。这种设计实现了高效的多帧集成和高效推理。此外,我们引入了一个新的基准测试,更有效地评估VLA模型的多帧集成能力。实验表明,DySta在我们的基准测试中各项指标上提高了24.5%的多帧集成能力,在真实世界记忆依赖任务中绝对成功率达到23.3%,同时在模拟基准测试中推理速度提升2.0倍(成功率+2.3%),在真实世界通用任务中推理速度提升2.2倍(成功率+10.6%)。
英文摘要:
Vision-Language-Action (VLA) models have recently emerged as a promising paradigm for generalist robotic control. Built upon vision-language model (VLM) architectures, VLAs predict actions conditioned on visual observations and language instructions, achieving strong performance and generalization across tasks. However, VLAs face two major challenges: a limited context window for input frames and inefficient inference due to the quadratic attention complexity and large parameter counts. To this end, we propose DySta, a framework that disentangles visual inputs into multi-level static and dynamic tokens, which enables (1) retaining a single copy of static tokens across frames to significantly reduce context length, and (2) reusing the key-value (KV) cache of static tokens through a lightweight recache gate that updates only when necessary. This design enables efficient multi-frame integration and efficient inference. In addition, we introduce a new benchmark that more effectively evaluates the multi-frame integration ability of VLAs. Experiments show that Dysta improves multi-frame integration by 24.5% across metrics on our benchmark and 23.3% in absolute success rate on real-world memory-dependent tasks, while accelerating inference by 2.0x (with +2.3% success rate) on simulation benchmarks and 2.2x (with +10.6% success rate) on real-world general tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。