VideoComp：推进视频-文本模型中的细粒度组合性与时序对齐
VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Models
AI总结：
本文提出 VideoComp 基准与学习框架，通过构建多事件视频组合性基准和分层成对偏好损失，提升视频-文本模型在细粒度时序对齐上的能力。
摘要：
我们引入了 VideoComp，一个用于推进视频-文本组合性理解的基准和学习框架，旨在提升视觉-语言模型（VLMs）在细粒度时序对齐方面的能力。与现有聚焦于静态图像-文本组合性或孤立单事件视频的基准不同，我们的基准针对连续多事件视频中的对齐。利用具有时序定位事件描述的视频-文本数据集（如 ActivityNet-Captions、YouCook2），我们构建了两个组合性基准：ActivityNet-Comp 和 YouCook2-Comp。我们创建了具有细微时序干扰的挑战性负样本，例如重排序、动作词替换、部分描述和组合干扰。这些基准全面测试模型在扩展的、连贯的视频-文本序列上的组合敏感性。为提升模型性能，我们提出了一种分层成对偏好损失，该损失强化与时序准确对的匹配，并逐步惩罚干扰程度递增的对，从而鼓励细粒度组合学习。为缓解密集标注视频数据有限的问题，我们引入了一种预训练策略，通过拼接短视频-描述对来模拟多事件序列。我们在基准上评估了视频-文本基础模型和大型多模态模型（LMMs），识别出其在组合性方面的优势与改进空间。总体而言，我们的工作提供了一个全面的框架，用于评估和增强模型在实现细粒度、时序连贯的视频-文本对齐方面的能力。
英文摘要：
We introduce VideoComp, a benchmark and learning framework for advancing video-text compositionality understanding, aimed at improving vision-language models (VLMs) in fine-grained temporal alignment. Unlike existing benchmarks focused on static image-text compositionality or isolated single-event videos, our benchmark targets alignment in continuous multi-event videos. Leveraging video-text datasets with temporally localized event captions (e.g. ActivityNet-Captions, YouCook2), we construct two compositional benchmarks, ActivityNet-Comp and YouCook2-Comp. We create challenging negative samples with subtle temporal disruptions such as reordering, action word replacement, partial captioning, and combined disruptions. These benchmarks comprehensively test models' compositional sensitivity across extended, cohesive video-text sequences. To improve model performance, we propose a hierarchical pairwise preference loss that strengthens alignment with temporally accurate pairs and gradually penalizes increasingly disrupted ones, encouraging fine-grained compositional learning. To mitigate the limited availability of densely annotated video data, we introduce a pretraining strategy that concatenates short video-caption pairs to simulate multi-event sequences. We evaluate video-text foundational models and large multimodal models (LMMs) on our benchmark, identifying both strengths and areas for improvement in compositionality. Overall, our work provides a comprehensive framework for evaluating and enhancing model capabilities in achieving fine-grained, temporally coherent video-text alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
