arXiv 2503.101112025-04-10cs.CV

基于帧融合与任务感知路由的持续文本到视频检索

Continual Text-to-Video Retrieval with Frame Fusion and Task-Aware Routing

The University of Queensland（昆士兰大学）；The University of Southern Queensland（南昆士兰大学）

Zecheng Zhao, Zhi Chen, Zi Huang, Shazia Sadiq, Tong Chen

备注：: Accepted at SIGIR 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出首个持续文本到视频检索基准，并设计FrameFusionMoE框架，通过帧融合适配器和任务感知专家混合机制，在适应新视频内容的同时缓解灾难性遗忘，实验表明其检索性能优于现有方法。

摘要：

文本到视频检索（TVR）旨在根据文本查询检索相关视频。然而，随着视频内容不断演变，使TVR系统适应新数据仍然是一个关键但尚未被充分探索的挑战。在本文中，我们提出了首个面向持续文本到视频检索（CTVR）的基准，以解决现有方法的局限性。当前基于预训练模型（PTM）的TVR方法在适应新任务时难以保持模型可塑性，而现有的持续学习（CL）方法则遭受灾难性遗忘，导致历史查询与已存储视频特征之间出现语义错位。为解决这两个挑战，我们提出了FrameFusionMoE，一种新颖的CTVR框架，包含两个关键组件：（1）帧融合适配器（FFA），用于捕捉视频时序动态，同时保持模型可塑性；（2）任务感知专家混合（TAME），确保跨任务查询与已存储视频特征之间保持一致的语义对齐。因此，FrameFusionMoE能够有效适应新的视频内容，同时保持历史文本-视频相关性，以缓解灾难性遗忘。我们在两个基准数据集上、多种任务设置下对FrameFusionMoE进行了全面评估。结果表明，FrameFusionMoE优于现有的CL和TVR方法，在处理连续视频流时取得了更优的检索性能，并且在早期任务上的性能退化最小。我们的代码可在以下地址获取：https://github.com/JasonCodeMaker/CTVR。

英文摘要：

Text-to-Video Retrieval (TVR) aims to retrieve relevant videos based on textual queries. However, as video content evolves continuously, adapting TVR systems to new data remains a critical yet under-explored challenge. In this paper, we introduce the first benchmark for Continual Text-to-Video Retrieval (CTVR) to address the limitations of existing approaches. Current Pre-Trained Model (PTM)-based TVR methods struggle with maintaining model plasticity when adapting to new tasks, while existing Continual Learning (CL) methods suffer from catastrophic forgetting, leading to semantic misalignment between historical queries and stored video features. To address these two challenges, we propose FrameFusionMoE, a novel CTVR framework that comprises two key components: (1) the Frame Fusion Adapter (FFA), which captures temporal video dynamics while preserving model plasticity, and (2) the Task-Aware Mixture-of-Experts (TAME), which ensures consistent semantic alignment between queries across tasks and the stored video features. Thus, FrameFusionMoE enables effective adaptation to new video content while preserving historical text-video relevance to mitigate catastrophic forgetting. We comprehensively evaluate FrameFusionMoE on two benchmark datasets under various task settings. Results demonstrate that FrameFusionMoE outperforms existing CL and TVR methods, achieving superior retrieval performance with minimal degradation on earlier tasks when handling continuous video streams. Our code is available at: https://github.com/JasonCodeMaker/CTVR.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。