快得足以行动：面向低延迟机器人视觉语言模型和视觉语言动作模型的时空视觉令牌融合
Fast Enough to Act: Spatio-Temporal Visual Token Merging for Low-Latency Robotic VLMs and VLAs
AI总结：
提出ST-Merge框架，在视觉编码阶段通过构建3D时空坐标和多队列并行匹配加权聚合机制高效融合冗余令牌，并引入后融合位置校正消除空间偏差，在Qwen2.5-VL上实现2倍推理加速且精度损失仅1%，在π0.5 VLA策略上实现8.3倍加速。
摘要：
视觉语言模型和视觉语言动作模型赋予了机器人前所未有的能力。然而，视频和高分辨率图像的输入会产生大量视觉令牌，导致极高的推理延迟，严重阻碍了机器人的实时控制。为了突破这一计算瓶颈，我们提出了ST-Merge，一个即插即用、无需训练的框架，直接在视觉编码阶段高效融合冗余令牌。通过显式构建3D时空坐标，它采用多队列并行匹配和加权聚合机制，实现跨帧冗余令牌的高效且几何一致的融合。此外，我们引入了一种后融合位置校正机制，通过动态重新评估视觉令牌加权质心的旋转位置编码，有效消除融合引起的空间偏差，从而确保灵巧操作所需的高精度空间感知。在主流VLM Qwen2.5-VL上的视频问答任务中，ST-Merge实现了2倍的推理加速，且精度仅损失1%。当部署在π0.5 VLA策略上时，ST-Merge在1024×1024分辨率下实现了8.3倍加速，并在此高分辨率设置下匹配了基线成功率。在较低分辨率下，它引入了少量精度下降。
英文摘要：
Vision-language models and vision-language action models endow the robot with unprecedented capabilities. However, the input of video and high-resolution images yields a massive number of visual tokens, leading to extremely high inference latency and severely hindering the robot's real-time control. To break through this computational bottleneck, we propose ST-Merge, a plug-and-play, training-free framework that efficiently fuses redundant tokens directly during the visual encoding phase. By explicitly constructing 3D spatiotemporal coordinates, it employs a multi-queue parallel matching and weighted aggregation mechanism to achieve efficient and geometrically consistent fusion of redundant tokens across frames. In addition, we introduce a post-merge positional correction mechanism that effectively eliminates spatial deviation caused by merging by dynamically re-evaluating the rotational position code of the weighted centroid of the vision token, thereby ensuring the high-precision spatial awareness required for dexterous operation. In the Video Question Answering task on the mainstream VLM, Qwen2.5-VL, ST-Merge achieves a 2$\times$ inference speedup with only a tiny 1\% loss in precision. When deployed on the $π_{0.5}$ VLA policy, ST-Merge achieves an 8.3$\times$ speedup at 1024 $\times$ 1024 resolution and matches the baseline success rate at this high-resolution setting. At lower resolutions, it introduces a small drop in accuracy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
