arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.08094cs.ROcs.AIcs.LGcs.SYeess.SY
收藏

vla.cpp:视觉-语言-动作模型的统一推理运行时

vla.cpp: A Unified Inference Runtime for Vision-Language-Action Models

VinRobotics;Center for AI Research, VinUniversity(VinUniversity 人工智能研究中心);Intelligent Autonomous Systems, TU Darmstadt(达姆施塔特工业大学智能自主系统);Max Planck Research School for Intelligent Systems(马克斯·普朗克智能系统研究学院);University of Stuttgart(斯图加特大学);German Research Center for Artificial Intelligence(德国人工智能研究中心)

Khanh D. Nguyen, Hung T. Ho, Chinh T. Nguyen, Thanh Q. Duong, Linh D. Le, Duy M. H. Nguyen, Vien A. Ngo, An T. Le

AI总结:

提出vla.cpp,基于llama.cpp的便携C++推理运行时,支持多种VLA架构,在LIBERO-Object上接近SOTA性能,内存仅1.3 GiB,并实现跨硬件部署。

摘要:

视觉-语言-动作(VLA)策略通常以Python/PyTorch堆栈形式提供,假设使用工作站级GPU,这与机器人实际运行的硬件不匹配。我们提出了vla.cpp,一个基于llama.cpp的便携式C++推理运行时。据我们所知,它是第一个原生支持流匹配和扩散VLA推理模式的ggml类引擎,其中缓存的视觉-语言前缀由交叉注意力动作专家在多个求解器步骤中消耗。单个运行时通过一个请求/响应协议服务于跨越五个骨干网络和四个动作头家族的七种架构,每个模型打包为自包含的捆绑包。在LIBERO-Object上,该引擎在200个回合中与最先进的检查点相差不到一个回合,并以1.3 GiB内存运行BitVLA达到100%成功率。相同的捆绑包在三个硬件层级上不变地运行,从消费级GPU到8 GB嵌入式模块。跨硬件屋顶线分析表明,批量大小为1的VLA推理受计算限制,因此利用率而非带宽是部署杠杆;由此分析得出的IMMA梯形GEMM将BitVLA每步延迟降低了4.5倍。然后,我们在ALOHA机械臂上设计了一个机载压力测试,隔离了学习型VLA必须在训练它的硬件上针对移动目标重新规划的延迟约束。代码、演示视频和可重复的基准测试框架可在https://fai-modelopt-tech.github.io/vla-cpp.github.io/获取。

英文摘要:

Deploying vision--language--action (VLA) models on robots requires adapting model-specific inference pipelines to heterogeneous processors and limited onboard memory. We present vla.cpp, a unified C++ inference runtime for eleven VLA models, with no PyTorch dependency for model execution. The runtime shares model loading, tensor execution, and serving while retaining architecture-specific attention, conditioning, and action heads. Iterative policies reuse observation-dependent computation across solver steps, while regression policies predict actions directly. We evaluate task success on LIBERO-Object and profile supported configurations on NVIDIA, Apple, and Intel hardware. BitVLA completes 200/200 LIBERO-Object episodes on an 8,GB Jetson Orin Nano. A ternary tensor-core kernel accelerates its client inference by $4.0$--$4.6\times$ over the CUDA-core baseline on RTX 3060 and AGX Orin. A SmolVLA case study links positional-index precision to gripper commands and task success, showing why fixed-input numerical checks should accompany rollout evaluation. Deployments on UR10e and ALOHA demonstrate physical robot integration; delay and execution-horizon studies characterize synchronous chunked control. The results demonstrate a common deployment path across VLA architectures and hardware, with numerical validation and control settings guiding deployment alongside inference efficiency.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑