arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.01558cs.CVcs.AI
收藏

VideoLights: 用于联合视频亮点检测和时刻检索的特征细化与跨任务对齐变换器

VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrieval

North South University(北南大学);Qatar Computing Research Institute (QCRI)(卡塔尔计算研究 institute)

Dhiman Paul, Md Rizwan Parvez, Nabeel Mohammed, Shafin Rahman

AI总结:

VideoLights通过引入特征细化、跨模态融合和联合任务反馈机制,提升视频亮点检测与时刻检索的性能。

摘要:

现有的联合预测变换器在处理跨任务动态、实现稳健的视频-文本对齐以及利用有效的注意力机制方面存在不足,大型语言/视觉-语言模型(LLMs/LVLMs)的潜力尚未被充分挖掘。本文介绍了VideoLights,一种新的HD/MR框架,通过引入:(i)带有对齐损失的卷积投影和特征细化模块以增强视频-文本特征一致性;(ii)双向跨模态融合网络以实现强耦合的查询感知表示;(iii)单向联合任务反馈机制以协同提升任务性能;(iv)硬正/负损失以实现自适应学习;以及(v)利用LVLMs(例如BLIP-2)以实现更优的多模态特征整合和智能预训练。在QVHighlights、TVSum和Charades-STA基准测试中,VideoLights显著优于现有基线,建立了新的最先进性能。代码和模型检查点可在https://github.com/dpaul06/VideoLights获取。

英文摘要:

Prevailing joint prediction transformers for Video Highlight Detection and Moment Retrieval (HD/MR) exhibit deficiencies in handling cross-task dynamics, achieving robust video-text alignment, and utilizing effective attention mechanisms, with the potential of Large Language/Vision-Language Models (LLMs/LVLMs) being largely untapped. This paper introduces VideoLights, a novel HD/MR framework addressing these limitations by incorporating: (i) Convolutional Projection and Feature Refinement modules with an alignment loss for enhanced video-text feature congruity; (ii) a Bi-Directional Cross-Modal Fusion network for strongly coupled query-aware representations; (iii) a Uni-directional joint-task feedback mechanism for synergistic task improvement; (iv) hard positive/negative losses for adaptive learning; and (v) the leveraging of LVLMs (e.g., BLIP-2) for superior multimodal feature integration and intelligent pre-training with synthetic data. Comprehensive evaluations on QVHighlights, TVSum, and Charades-STA benchmarks demonstrate that VideoLights significantly surpasses existing baselines, establishing new state-of-the-art performances. Codes and model checkpoints are available at https://github.com/dpaul06/VideoLights .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑