从语言输入角度重新思考视频-语言模型
Rethinking Video-Language Model from the Language Input Perspective
AI总结：
本文从语言输入角度出发，提出一种即插即用的框架，通过生成正负文本、属性文本推理和自加权损失，提升视频-语言模型的性能。
摘要：
受大语言模型浪潮的驱动，视频-语言模型（VLM）已成为弥合视频与文本之间差距的重要且具有挑战性的技术。尽管先前的VLM工作取得了显著进展，但几乎所有工作都隐含地假设所有文本都是由特定模板预定义的。在实际应用中，这种严格的假设无法满足，因为1）预定义所有文本极其耗时费力；2）这些预定义的文本输入过于限制且不友好，限制了其应用。观察到，给定视频输入，语义相似但模板不同的文本会导致不同的性能。为此，本文提出了一种新颖的即插即用框架，用于各种基于VLM的方法，以充分弥合视频和文本。具体来说，我们首先从原始文本中生成正负文本，以针对特定的文本组件。然后，我们提出了一种基于属性的文本推理策略，以挖掘生成文本的细粒度语义。最后，我们利用视频作为指导，通过设计自加权损失来进行跨模态桥接。大量实验表明，所提出的方法可以作为即插即用模块，有效提升最先进VLM的性能。
英文摘要：
Driven by the wave of large language models, Video-Language Models (VLMs) have become a significant yet challenging technology to bridge the gap between videos and texts. Although previous VLM works have made significant progress, almost all of them implicitly assume that all the texts are predefined by the specific template. In real-world applications, such a strict assumption is impossible to satisfy since 1) predefining all the texts is extremely time-consuming and labor-intensive. 2) these predefined text inputs are too restrictive and user-unfriendly, limiting their applications. It is observed that given a video input, texts with similar semantics but different templates lead to various performances. To this end, in this paper, we propose a novel plug-and-play framework for various VLM-based methods to fully bridge videos and texts. Specifically, we first generate positive and negative texts from the original ones to target specific text components. Then, we propose an attribute-based text reasoning strategy to mine fine-grained textual semantics of generated texts. Finally, we utilize videos as guidance to conduct cross-modal bridging by designing a self-weighted loss. Extensive experiments show that the proposed method can serve as the plug-and-play module to effectively improve the performance of state-of-the-art VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
