arXiv 2508.015922026-03-03cs.CVcs.AI

DMTrack: 基于双适配器的时空多模态跟踪

DMTrack: Spatio-Temporal Multimodal Tracking via Dual-Adapter

Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（中国科学院大学杭州高等研究院）；Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）；Department of Computer Science and Engineering, University of North Texas（北卡罗来纳州立大学计算机科学与工程系）；OPPO AI Center（OPPO人工智能中心）

Weihong Li, Shaohua Dong, Haonan Lu, Yanhao Zhang, Heng Fan, Libo Zhang

备注：: Accepted by ICRA 2026

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

DMTrack通过双适配器架构实现时空多模态跟踪，利用STMA和PMCA模块提升跨模态融合效果，达到先进性能。

摘要：

本文探讨了适配器调优技术，并引入了一种新颖的双适配器架构，称为DMTrack，用于时空多模态跟踪。DMTrack的核心在于两个简单而有效的模块，包括一个时空模态适配器（STMA）和一个渐进模态互补适配器（PMCA）模块。前者应用于每个模态单独，旨在通过自我提示调整冻结主干网络提取的时空特征，从而在一定程度上弥合不同模态之间的差距，从而实现更好的跨模态融合。后者旨在通过两个特别设计的像素级浅层和深层适配器逐步促进跨模态提示。浅层适配器在两个模态之间共享参数，旨在弥合两个模态分支之间的信息流，从而为后续的模态融合奠定基础；而深层适配器则通过像素级内部模态注意力调节初步融合的信息流，并进一步通过像素级跨模态注意力生成模态感知的提示。通过这样的设计，DMTrack仅使用0.93M可训练参数就实现了有希望的时空多模态跟踪性能。在五个基准测试中的广泛实验表明，DMTrack取得了最先进的结果。我们的代码和模型将在https://github.com/Nightwatch-Fox11/DMTrack上提供。

英文摘要：

In this paper, we explore adapter tuning and introduce a novel dual-adapter architecture for spatio-temporal multimodal tracking, dubbed DMTrack. The key of our DMTrack lies in two simple yet effective modules, including a spatio-temporal modality adapter (STMA) and a progressive modality complementary adapter (PMCA) module. The former, applied to each modality alone, aims to adjust spatio-temporal features extracted from a frozen backbone by self-prompting, which to some extent can bridge the gap between different modalities and thus allows better cross-modality fusion. The latter seeks to facilitate cross-modality prompting progressively with two specially designed pixel-wise shallow and deep adapters. The shallow adapter employs shared parameters between the two modalities, aiming to bridge the information flow between the two modality branches, thereby laying the foundation for following modality fusion, while the deep adapter modulates the preliminarily fused information flow with pixel-wise inner-modal attention and further generates modality-aware prompts through pixel-wise inter-modal attention. With such designs, DMTrack achieves promising spatio-temporal multimodal tracking performance with merely 0.93M trainable parameters. Extensive experiments on five benchmarks demonstrate that DMTrack achieves state-of-the-art results. Our code and models will be available at https://github.com/Nightwatch-Fox11/DMTrack.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。