Computer Vision (CV) has yet to fully achieve the zero-shot task generalization observed in Natural Language Processing (NLP), despite following many of the milestones established in NLP, such as large transformer models, extensive pre-training, and the auto-regression paradigm, among others. In this paper, we explore the idea that CV adopts discrete and terminological task definitions (\eg, ``image segmentation''), which may be a key barrier to zero-shot task generalization. Our hypothesis is that without truly understanding previously-seen tasks--due to these terminological definitions--deep models struggle to generalize to novel tasks. To verify this, we introduce Explanatory Instructions, which provide an intuitive way to define CV task objectives through detailed linguistic transformations from input images to outputs. We create a large-scale dataset comprising 12 million ``image input $\to$ explanatory instruction $\to$ output'' triplets, and train an auto-regressive-based vision-language model (AR-based VLM) that takes both images and explanatory instructions as input. By learning to follow these instructions, the AR-based VLM achieves instruction-level zero-shot capabilities for previously-seen tasks and demonstrates strong zero-shot generalization for unseen CV tasks. Code and dataset will be openly available on our GitHub repository.
Beihang University（北京航空航天大学）；Xiaohongshu Inc（小红书公司）；Shenzhen Institute of Advanced Technology（深圳先进技术研究院）；Nanchang Institute of Technology（南昌工程学院）；Zhongguancun Laboratory（中关村实验室）
Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning
School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）
University of Maryland, College Park（马里兰大学帕克分校）；Amazon Alexa AI（亚马逊Alexa人工智能部门）
Household environments are visually diverse. Embodied agents performing Vision-and-Language Navigation (VLN) in the wild must be able to handle this diversity, while also following arbitrary language instructions. Recently, Vision-Language models like CLIP have shown great performance on the task of zero-shot object recognition. In this work, we ask if these models are also capable of zero-shot language grounding. In particular, we utilize CLIP to tackle the novel problem of zero-shot VLN using natural language referring expressions that describe target objects, in contrast to past work that u…
University of Bonn（波恩大学）；National University of Singapore（新加坡国立大学）
How can we teach a robot to predict what will happen next for an activity it has never seen before? We address this problem of zero-shot anticipation by presenting a hierarchical model that generalizes instructional knowledge from large-scale text-corpora and transfers the knowledge to the visual domain. Given a portion of an instructional video, our model predicts coherent and plausible actions multiple steps into the future, all in rich natural language. To demonstrate the anticipation capabilities of our model, we introduce the Tasty Videos dataset, a collection of 2511 recipes for zero-sho…
We study the problem of learning a robot policy to follow natural language instructions that can be easily extended to reason about new objects. We introduce a few-shot language-conditioned object grounding method trained from augmented reality data that uses exemplars to identify objects and align them to their mentions in instructions. We present a learned map representation that encodes object locations and their instructed use, and construct it from our few-shot grounding output. We integrate this mapping approach into an instruction-following policy, thereby allowing it to reason about pr…
University of Bonn（波恩大学）；Indian Institute of Technology (ISM) Dhanbad（印度理工学院（印度矿业学院）丹巴德分校）；National University of Singapore（新加坡国立大学）
Can we teach a robot to recognize and make predictions for activities that it has never seen before? We tackle this problem by learning models for video from text. This paper presents a hierarchical model that generalizes instructional knowledge from large-scale text corpora and transfers the knowledge to video. Given a portion of an instructional video, our model recognizes and predicts coherent and plausible actions multiple steps into the future, all in rich natural language. To demonstrate the capabilities of our model, we introduce the \emph{Tasty Videos Dataset V2}, a collection of 4022 …
Constraint-Aware Zero-Shot Vision-Language Navigation in Continuous Environments
New Laboratory of Pattern Recognition (NLPR)（模式识别新实验室）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)（穆罕默德·本·扎耶德人工智能大学）；Robotics X, Tencent（腾讯Robotics X实验室）
Socratic Planner: Self-QA-Based Zero-Shot Planning for Embodied Instruction Following
Seoul National University（首尔大学）；Interdisciplinary Program in AI (IPAI), Seoul National University（首尔大学人工智能交叉项目）；AI Institute, Seoul National University（首尔大学人工智能研究院）
$A^2$Nav: Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models
South China University of Technology（华南理工大学）；Shenzhen University（深圳大学）；Pazhou Laboratory（琶洲实验室）；Key Laboratory of Big Data and Intelligent Robot, Ministry of Education（教育部大数据与智能机器人重点实验室）；Shenzhen Graduate School, Peking University（北京大学深圳研究生院）；UMass Amherst（马萨诸塞大学阿默斯特分校）；Cisco Research（思科研究院）
Open-Nav: Exploring Zero-Shot Vision-and-Language Navigation in Continuous Environment with Open-Source LLMs
Australian Institute for Machine Learning at the University of Adelaide（阿德莱德大学澳大利亚机器学习研究所）；School of Software Engineering, South China University of Technology（华南理工大学软件工程学院）
Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models
Institute of Image Processing and Pattern Recognition（图像处理与模式识别研究所）；Institute of Medical Robotics（医学机器人研究所）；Shanghai Jiao Tong University（上海交通大学）；Tencent YouTu Lab（腾讯优图实验室）
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Zhejiang Normal University（浙江师范大学）
Vision-and-Language Navigation in Continuous Environments (VLN-CE), which links language instructions to perception and control in the real world, is a core capability of embodied robots. Recently, large-scale pretrained foundation models have been leveraged as shared priors for perception, reasoning, and action, enabling zero-shot VLN without task-specific training. However, existing zero-shot VLN methods depend on costly perception and passive scene understanding, collapsing control to point-level choices. As a result, they are expensive to deploy, misaligned in action semantics, and short-s…
