$Δ$VLA: Prior-Guided Vision-Language-Action Models via World Knowledge Variation
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Great Bay University(大湾大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Macao Polytechnic University(澳门理工学院)
Yijie Zhu, Jie He, Rui Shao, Kaishen Yuan, Tao Tan, Xiaochen Yuan, Zitong Yu
最近的视觉-语言-动作(VLA)模型通过统一感知、推理和控制显著推动了机器人操作的发展。为了实现这种整合,最近的研究采用了一种预测范式,通过建模未来的视觉状态或世界知识来指导动作生成。然而,这些模型强调预测结果而非推理变化过程的本质,这对于确定如何行动至关重要。为此,我们提出了$Δ$VLA,一种基于先验引导的框架,通过相对于显式当前世界知识先验建模世界知识变化来生成动作,而不是回归绝对的未来世界状态。具体而言,1)为了构建当前世界知识先验,我们提出了Prior-Guided WorldKnowledge Extractor(PWKE)。它从视觉输入中提取可操作区域、空间关系和语义线索,受辅助头和先验伪标签的引导,从而减少冗余。2)在此基础上,为了表示在动作下世界知识如何演变,我们引入了Latent World Variation Quantization(LWVQ)。它通过VQ-VAE目标学习离散的潜在空间来编码世界知识变化,将预测从全模态转移到紧凑的潜在空间。3)此外,为了在变化建模过程中减少干扰,我们设计了Conditional Variation Attention(CV-Atten),它促进了解耦学习并保持了知识表示的独立性。在模拟基准和真实世界机器人任务上的广泛实验表明,$Δ$VLA实现了最先进的性能,同时提高了效率。代码和真实世界执行视频可在https://github.com/JiuTian-VL/DeltaVLA上获得。
英文摘要:
Recent vision-language-action (VLA) models have significantly advanced robotic manipulation by unifying perception, reasoning, and control. To achieve such integration, recent studies adopt a predictive paradigm that models future visual states or world knowledge to guide action generation. However, these models emphasize forecasting outcomes rather than reasoning about the underlying process of change, which is essential for determining how to act. To address this, we propose $Δ$VLA, a prior-guided framework that models world-knowledge variations relative to an explicit current-world knowledge prior for action generation, rather than regressing absolute future world states. Specifically, 1) to construct the current world knowledge prior, we propose the Prior-Guided WorldKnowledge Extractor (PWKE). It extracts manipulable regions, spatial relations, and semantic cues from the visual input, guided by auxiliary heads and prior pseudo labels, thus reducing redundancy. 2) Building upon this, to represent how world knowledge evolves under actions, we introduce the Latent World Variation Quantization (LWVQ). It learns a discrete latent space via a VQ-VAE objective to encode world knowledge variations, shifting prediction from full modalities to compact latent. 3)Moreover, to mitigate interference during variation modeling, we design the Conditional Variation Attention (CV-Atten), whichpromotes disentangled learning and preserves the independence of knowledge representations. Extensive experiments on both simulated benchmarks and real-world robotic tasks demonstrate $Δ$VLA achieves state-of-the-art performance while improving efficiency. Code and real-world execution videos are available at https://github.com/JiuTian-VL/DeltaVLA.
Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoning
University of Science and Technology of China(中国科学技术大学);Shanghai AI Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);Zhejiang University(浙江大学);Nanjing University(南京大学);Fudan University(复旦大学);Tsinghua University(清华大学);NUS(新加坡国立大学);Northeastern University(东北大学);Shenzhen University(深圳大学)
Look Before Acting: Enhancing Vision Foundation Representations for Vision-Language-Action Models
State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(多媒体信息处理国家重点实验室,计算机学院,北京大学);Simplexity Robotics(Simplexity机器人公司);The Chinese University of Hong Kong(香港中文大学)
VLA-4D: Embedding 4D Awareness into Vision-Language-Action Models for SpatioTemporally Coherent Robotic Manipulation
School of Computing, National University of Singapore(新加坡国立大学计算机学院);School of Artificial Intelligence and Automation, Huazhong University of Science and Technology(华中科技大学人工智能与自动化学院)
QDepth-VLA: Quantized Depth Prediction as Auxiliary Supervision for Vision-Language-Action Models
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Beijing Zhongke Huiling Robot Technology Co.(北京中科创联机器人科技有限公司)
Reasoning-VLA: A Fast and General Vision-Language-Action Reasoning Model for Autonomous Driving
Lanzhou University(兰州大学);National University of Singapore(新加坡国立大学);University of Science and Technology of China(中国科学技术大学);Tsinghua University(清华大学);University of New South Wales(新南威尔士大学)
MAP-VLA: Memory-Augmented Prompting for Vision-Language-Action Model in Robotic Manipulation
Nanyang Technological University(南洋理工大学);VinUniversity(文大学);Beijing University of Posts and Telecommunications(北京邮电大学);Tsinghua University(清华大学);South China University of Technology(华南理工大学)
EmbodiedMidtrain: Bridging the Gap between Vision-Language Models and Vision-Language-Action Models via Mid-training
Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所);Bosch Research North America & Bosch Center for Artificial Intelligence (BCAI)(博世北美研究部及博世人工智能中心(BCAI))
VLADriver-RAG: Retrieval-Augmented Vision-Language-Action Models for Autonomous Driving
College of Automotive Engineering(汽车工程学院);The National Key Laboratory of Automotive Chassis Integration and Bionics(汽车底盘集成与生物力学国家级重点实验室);ReeFocus AI Technology(ReeFocus人工智能技术)
VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulators
Westlake University(西湖大学);Zhejiang University(浙江大学);OpenHelix Team(OpenHelix团队);Fudan University(复旦大学);Zhengzhou University(郑州大学);BUPT(北京邮电大学);Hebei University of Technology(河北工业大学)
VisualThink-VLA: Visual Intermediate Reasoning for Effective and Low-Latency Vision-Language-Action Policies
Zhejiang University(浙江大学);Cornell University(康奈尔大学);National University of Singapore(新加坡国立大学);Xi'an University of Electronic Science and Technology(西安电子科技大学)
VLA-JEPA: Enhancing Vision-Language-Action Model with Latent World Model
University of Science and Technology of China(中国科学技术大学);Zhongguancun Academy, Beijing, China(中关村学院);Shanghai Jiao Tong University(上海交通大学);Tsinghua University(清华大学);Eastern Institute of Technology, Ningbo(宁波东部科技研究院);University of Chinese Academy of Sciences(中国科学院大学);Nankai University(南开大学)
ActDistill: General Action-Guided Self-Derived Distillation for Efficient Vision-Language-Action Models
School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院);University of Technology Sydney(悉尼科技大学);Advanced Institute of Big Data(大数据先进研究院)
ForgeVLA: Federated Vision-Language-Action Learning without Language Annotations
Sichuan University(四川大学);Zhejiang University(浙江大学);Ludwig-Maximilians-Universität München(慕尼黑路德维希-马克西米利安大学);Lenovo Group Limited(联想集团有限公司);Engineering Research Center of Machine Learning and Industry Intelligence, Ministry of Education(教育部机器学习与工业智能工程研究中心)