FUTURE-VLA:在实时执行下统一轨迹预测
PACT-WAM: Predicting Actions and Visual Foresight with Compact Temporal Encoding for Robot Manipulation
AI总结:
FUTURE-VLA通过实时执行实现统一轨迹预测,采用双效效率范式提升时空信息密度,实现实时预测与人机交互机制,取得多项任务的高成功率。
摘要:
通用视觉-语言模型越来越多地支持对长视频流的统一时空推理,但将其部署在机器人上仍然受到处理长时间跨度历史和生成高维未来预测的延迟的限制。为弥合这一差距,我们提出了FUTURE-VLA,一种统一的架构,将长时间跨度的控制和未来预测重新表述为一个整体的序列生成任务。采用双效效率范式,FUTURE-VLA利用时间自适应压缩策略,最大限度地提高时空信息密度,使能够处理大量多视角历史的同时保持恒定的推理延迟。同时,它在潜在空间中进行自回归,使可操作的动力学与可审查的视觉前瞻在单次正向传递中对齐。这些实时预测能力进一步使预测引导的人机交互机制通过交互执行门控得以实现,允许操作员根据可解释的未来预览动态验证行为。广泛的评估表明,FUTURE-VLA建立了新的最先进性能,在LIBERO上达到99.2%的成功率,在RoboTwin上达到75.4%的成功率,在现实世界的Piper平台上达到78.0%的成功率,所有这些都在扩展了16倍的时空窗口的同时保持单帧基线的推理延迟。
英文摘要:
Robot manipulation uses temporal context to select actions and visual foresight to assess their consequences, yet dense representations of past and future observations incur substantial processing costs. We introduce PACT-WAM, a world-action model that jointly generates a 16-step action trajectory and its temporally corresponding visual forecast through conditional flow sampling. Hierarchical history encoding assigns coarse spatial representations to earlier observations and finer representations to recent ones, retaining 16 observations with 256 tokens per view, 75% fewer than dense encoding of the same frames. A shared flow module jointly updates continuous action and visual states through two modality-specific heads under transition-wise causal attention, and a TiTok-VAE decoder reconstructs multi-view future images from the visual latents. Decoded forecasts also support Proposal Review (PR), a vision-language model component for execution-prefix selection and proposal rejection. Without PR, PACT-WAM achieves average success rates of 98.6%, 92.3%, and 78.0% on LIBERO, RoboTwin 2.0, and real-world Piper tasks, respectively. PR provides a test-time enhancement, raising these rates to 99.5%, 93.4%, and 86.7%. Ablations show that hierarchical history allocation and joint action-visual generation improve control success, while analyses of visual capacity and forecast-guided execution characterize the trade-offs between success and proposal-generation cost.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。