arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.19975cs.CV
收藏

面向自然语言跟踪的语言与视觉参考的上下文感知集成

Context-Aware Integration of Language and Visual References for Natural Language Tracking

Zhejiang University of Technology(浙江工业大学);Nanyang Technological University(南洋理工大学);Zhejiang University(浙江大学);The Hong Kong University of Science and Technology(香港科技大学)

Yanyan Shao, Shuting He, Qi Ye, Yuchao Feng, Wenhan Luo, Jiming Chen

AI总结:

针对自然语言跟踪中语言与视觉模板不匹配及后期合并歧义导致的漂移问题,提出联合多模态跟踪框架,通过提示调制和统一目标解码实现端到端预测,并在多个基准上取得有竞争力的跟踪与定位性能。

摘要:

基于自然语言描述的跟踪(TNL)旨在给定初始帧中的语言描述时,在视频序列中持续定位目标。现有方法分别执行基于语言的匹配和基于模板的匹配来进行目标推理,并将来自两个来源的匹配结果进行合并,这在语言和视觉模板与动态目标状态不一致以及后续合并阶段存在歧义时会导致跟踪漂移。为了解决这些问题,我们提出了一种联合多模态跟踪框架,该框架包含:1)一个提示调制模块,用于利用时序视觉模板与语言表达之间的互补性,从而获得精确且上下文感知的外观线索和语言线索;2)一个统一的目标解码模块,用于集成多模态参考线索,并在搜索图像上执行集成后的查询,以端到端的方式直接预测目标位置。该设计通过利用历史视觉信息来确保时空一致性,并引入了一种集成式解决方案,在单一步骤中生成预测。在 TNL2K、OTB-Lang、LaSOT 和 RefCOCOg 上进行的大量实验验证了我们所提出方法的有效性。结果表明,在跟踪和定位任务上,该方法均取得了与最先进方法相比具有竞争力的性能。

英文摘要:

Tracking by natural language specification (TNL) aims to consistently localize a target in a video sequence given a linguistic description in the initial frame. Existing methodologies perform language-based and template-based matching for target reasoning separately and merge the matching results from two sources, which suffer from tracking drift when language and visual templates miss-align with the dynamic target state and ambiguity in the later merging stage. To tackle the issues, we propose a joint multi-modal tracking framework with 1) a prompt modulation module to leverage the complementarity between temporal visual templates and language expressions, enabling precise and context-aware appearance and linguistic cues, and 2) a unified target decoding module to integrate the multi-modal reference cues and executes the integrated queries on the search image to predict the target location in an end-to-end manner directly. This design ensures spatio-temporal consistency by leveraging historical visual information and introduces an integrated solution, generating predictions in a single step. Extensive experiments conducted on TNL2K, OTB-Lang, LaSOT, and RefCOCOg validate the efficacy of our proposed approach. The results demonstrate competitive performance against state-of-the-art methods for both tracking and grounding.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑