arXiv 2506.176292026-05-25cs.CVcs.AIcs.CL

CLiViS: 通过语言-视觉协同释放认知地图用于具身视觉推理

CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoning

School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；King Abdullah University of Science and Technology（科廷大学）；Fudan University（复旦大学）

Kailing Li, Qi'ao Xu, Tianwen Qian, Yuqian Fu, Yang Jiao, Xiaoling Wang

发表信息：: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 5134-5143

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出CLiViS框架，通过LLM进行高层任务规划并协调VLM驱动的开放世界视觉感知，构建动态认知地图以迭代更新场景上下文，实现无需训练的具身视觉推理。

摘要：

具身视觉推理（EVR）旨在基于自我中心视频遵循复杂、自由形式的指令，从而在动态环境中实现语义理解和时空推理。尽管具有潜力，EVR面临复杂指令多样性和长期自我中心视频中复杂时空动态的挑战。现有解决方案要么在静态视频描述上使用大型语言模型（LLM），这通常会遗漏关键视觉细节，要么依赖端到端视觉语言模型（VLM），后者在逐步组合推理上存在困难。考虑到LLM在推理和VLM在感知方面的互补优势，我们提出了CLiViS。这是一个新颖的无训练框架，利用LLM进行高层任务规划，并协调VLM驱动的开放世界视觉感知，以迭代更新场景上下文。基于这种协同，CLiViS的核心是一个动态认知地图，它在推理过程中不断演化。该地图构建了具身场景的结构化表示，连接了低层感知和高层推理。跨多个基准的大量实验证明了CLiViS的有效性和通用性，特别是在处理长期视觉依赖方面。代码可在 https://github.com/Teacher-Tom/CLiViS 获取。

英文摘要：

Embodied Visual Reasoning (EVR) seeks to follow complex, free-form instructions based on egocentric video, enabling semantic understanding and spatiotemporal reasoning in dynamic environments. Despite its promising potential, EVR encounters significant challenges stemming from the diversity of complex instructions and the intricate spatiotemporal dynamics in long-term egocentric videos. Prior solutions either employ Large Language Models (LLMs) over static video captions, which often omit critical visual details, or rely on end-to-end Vision-Language Models (VLMs) that struggle with stepwise compositional reasoning. Consider the complementary strengths of LLMs in reasoning and VLMs in perception, we propose CLiViS. It is a novel training-free framework that leverages LLMs for high-level task planning and orchestrates VLM-driven open-world visual perception to iteratively update the scene context. Building on this synergy, the core of CLiViS is a dynamic Cognitive Map that evolves throughout the reasoning process. This map constructs a structured representation of the embodied scene, bridging low-level perception and high-level reasoning. Extensive experiments across multiple benchmarks demonstrate the effectiveness and generality of CLiViS, especially in handling long-term visual dependencies. Code is available at https://github.com/Teacher-Tom/CLiViS.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。