VTD-CLIP：通过提示CLIP实现视频到文本的离散化
VTD-CLIP: Video-to-Text Discretization via Prompting CLIP
AI总结：
提出VTD-CLIP框架，利用冻结文本编码器构建视觉码本，将视频转为文本标记，并引入置信度感知融合和可学习提示，在多个视频基准上超越现有方法。
摘要：
视觉-语言模型弥合了视觉与语言理解之间的鸿沟，并已被证明在视频识别任务中具有强大的能力。现有方法主要依赖于对图像-文本预训练模型进行参数高效的微调，但由于时间建模不足，它们常常面临可解释性有限和泛化能力差的问题。为解决这些问题，我们提出了一种简单而有效的视频到文本离散化框架。我们的方法重新利用冻结的文本编码器，基于多模态预训练中视觉与文本嵌入之间的多对一对比对齐，从视频类别标签构建视觉码本。该码本通过特征查找有效地将时间视觉数据转换为文本标记，并通过显式视频建模提供可解释的视频表示。然后，为了增强对无关或噪声帧的鲁棒性，我们引入了一个置信度感知融合模块，该模块通过码本评估关键帧的语义相关性，动态加权关键帧。此外，我们的方法结合了可学习的文本提示，以进行自适应的码本更新。在HMDB-51、UCF-101、SSv2和Kinetics-400上的大量实验验证了我们方法的优越性，相比最先进的方法取得了更具竞争力的改进。代码将在https://github.com/isxinxin/VTD-CLIP公开提供。
英文摘要：
Vision-language models bridge visual and linguistic understanding and have proven to be powerful for video recognition tasks. Existing approaches primarily rely on parameter-efficient fine-tuning of image-text pre-trained models, yet they often suffer from limited interpretability and poor generalization due to inadequate temporal modeling. To address these, we propose a simple yet effective video-to-text discretization framework. Our method repurposes the frozen text encoder to construct a visual codebook from video class labels due to the many-to-one contrastive alignment between visual and textual embeddings in multimodal pretraining. This codebook effectively transforms temporal visual data into textual tokens via feature lookups and offers interpretable video representations through explicit video modeling. Then, to enhance robustness against irrelevant or noisy frames, we introduce a confidence-aware fusion module that dynamically weights keyframes by assessing their semantic relevance via the codebook. Furthermore, our method incorporates learnable text prompts to conduct adaptive codebook updates. Extensive experiments on HMDB-51, UCF-101, SSv2, and Kinetics-400 have validated the superiority of our approach, achieving more competitive improvements over state-of-the-art methods. The code will be publicly available at https://github.com/isxinxin/VTD-CLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
