NAVERO:解锁视频-语言组合性的细粒度语义
NAVERO: Unlocking Fine-Grained Semantics for Video-Language Compositionality
AI总结:
针对视频中组合关系随时间快速变化导致VidL模型组合理解困难的问题,该研究构建AARO基准,提出负样本增强的NAVERO训练方法及匹配损失,在组合理解和检索任务上均取得更优表现。
摘要:
我们研究了视频-语言(VidL)模型对物体、属性、动作及其关系之间组合关系的理解能力。由于视频中的组合关系会随时间快速变化,组合理解对视频数据而言极具挑战性。我们首先构建了名为AARO的基准,用于评估基于空间概念的动作相关组合理解能力。该基准通过为给定视频生成带有错误动作描述的负样本文本构建,要求模型将正样本与其对应视频匹配。此外,我们提出了名为NAVERO的训练方法,利用经负样本文本增强的视频-文本数据提升组合理解能力。我们还设计了负样本增强的视觉-语言匹配损失,专门用于从生成的负样本文本中获益。我们在组合理解能力和视频-文本检索性能两方面,将NAVERO与其他前沿方法进行对比。NAVERO在视频-语言和图像-语言组合理解上均较其他方法取得显著提升,同时在传统文本-视频检索任务上保持了优异性能。
英文摘要:
We study the capability of Video-Language (VidL) models in understanding compositions between objects, attributes, actions and their relations. Composition understanding becomes particularly challenging for video data since the compositional relations rapidly change over time in videos. We first build a benchmark named AARO to evaluate composition understanding related to actions on top of spatial concepts. The benchmark is constructed by generating negative texts with incorrect action descriptions for a given video and the model is expected to pair a positive text with its corresponding video. Furthermore, we propose a training method called NAVERO which utilizes video-text data augmented with negative texts to enhance composition understanding. We also develop a negative-augmented visual-language matching loss which is used explicitly to benefit from the generated negative text. We compare NAVERO with other state-of-the-art methods in terms of compositional understanding as well as video-text retrieval performance. NAVERO achieves significant improvement over other methods for both video-language and image-language composition understanding, while maintaining strong performance on traditional text-video retrieval tasks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。