arXiv 2505.207182025-05-28cs.CVcs.AI

VLM可以成为好助手：利用自我改进的视觉语言模型增强具身视觉跟踪

VLM Can Be a Good Assistant: Enhancing Embodied Visual Tracking with Self-Improving Vision-Language Models

State Key Laboratory of Complex & Critical Software Environment, Beihang University（北京航空航天大学复杂与关键软件环境国家重点实验室）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）；City University of Macau（澳门城市大学）；Center on Frontiers of Computing Studies, School of Computer Science, Nat’l Eng. Research Center of Visual Technology, Peking University（北京大学计算机学院前沿计算研究中心、国家工程视觉技术研究中心）

Kui Wu, Shuhang Xu, Hao Chen, Churan Wang, Zhoujun Li, Yizhou Wang, Fangwei Zhong

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出一种自我改进框架，结合VLM推理与主动跟踪方法，通过记忆增强机制解决3D空间推理局限，显著提升具身视觉跟踪在失败后的恢复能力。

摘要：

我们引入了一种新颖的自我改进框架，该框架利用视觉语言模型（VLMs）来增强具身视觉跟踪（EVT），以解决当前主动视觉跟踪系统在从跟踪失败中恢复方面的局限性。我们的方法将现成的主动跟踪方法与VLM的推理能力相结合，在正常跟踪时部署快速的视觉策略，仅在检测到失败时才激活VLM推理。该框架具有记忆增强的自我反思机制，使VLM能够通过从过去的经验中学习来逐步改进，有效解决了VLM在3D空间推理方面的局限性。实验结果表明，性能得到了显著提升，在具有挑战性的环境中，我们的框架使基于强化学习（RL）方法的成功率提高了72%，使基于PID方法的成功率提高了220%。这项工作首次整合了基于VLM的推理来协助EVT智能体进行主动的失败恢复，为需要在动态、非结构化环境中持续监测目标的现实世界机器人应用提供了实质性的进展。项目网站：https://sites.google.com/view/evt-recovery-assistant。

英文摘要：

We introduce a novel self-improving framework that enhances Embodied Visual Tracking (EVT) with Vision-Language Models (VLMs) to address the limitations of current active visual tracking systems in recovering from tracking failure. Our approach combines the off-the-shelf active tracking methods with VLMs' reasoning capabilities, deploying a fast visual policy for normal tracking and activating VLM reasoning only upon failure detection. The framework features a memory-augmented self-reflection mechanism that enables the VLM to progressively improve by learning from past experiences, effectively addressing VLMs' limitations in 3D spatial reasoning. Experimental results demonstrate significant performance improvements, with our framework boosting success rates by $72\%$ with state-of-the-art RL-based approaches and $220\%$ with PID-based methods in challenging environments. This work represents the first integration of VLM-based reasoning to assist EVT agents in proactive failure recovery, offering substantial advances for real-world robotic applications that require continuous target monitoring in dynamic, unstructured environments. Project website: https://sites.google.com/view/evt-recovery-assistant.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。