TOPA：通过纯文本预对齐扩展大语言模型以实现视频理解
TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment
AI总结：
针对视频理解中语言监督效率低的问题，提出TOPA纯文本预对齐方法，通过生成文本视频预对齐LLM，无需真实视频预训练即可在Egoschema基准上取得51.0%的Top-1准确率，性能优于传统预训练方法。
摘要：
图像理解领域的近期进展得益于网络图像-文本对的广泛应用。然而，尽管存在大量网络视频-文本数据，视频理解仍是一项挑战。这一困难主要源于视频固有的复杂性，以及近期网络收集的视频-文本数据集中语言监督效率低下的问题。本文提出了纯文本预对齐（Text-Only Pre-Alignment, TOPA）方法，这是一种无需在真实视频数据上进行预训练即可扩展大语言模型（LLMs）用于视频理解的新方法。具体而言，我们首先利用先进的LLM自动生成包含连续文本帧的“文本视频”，并生成相应标注来模拟真实视频-文本数据。随后，这些带标注的文本视频被用于对纯语言LLM进行视频模态的预对齐。为弥合文本视频与真实视频之间的差距，我们采用CLIP模型作为特征提取器来对齐图像与文本模态。在纯文本预对齐过程中，编码为CLIP文本特征序列的连续文本帧与连续CLIP图像特征具有相似性，从而使LLM与真实视频表示对齐。大量实验（包括零样本评估和在多种视频理解任务上的微调）表明，TOPA是一种有效且高效的视频内容与LLM对齐框架。值得注意的是，在未使用任何视频数据训练的情况下，TOPA-Llama2-13B模型在具有挑战性的长视频理解基准Egoschema上取得了51.0%的Top-1准确率，该性能超越了以往的视频-文本预训练方法，且与近期基于GPT-3.5的视频智能体相比具有竞争力。
英文摘要：
Recent advancements in image understanding have benefited from the extensive use of web image-text pairs. However, video understanding remains a challenge despite the availability of substantial web video-text data. This difficulty primarily arises from the inherent complexity of videos and the inefficient language supervision in recent web-collected video-text datasets. In this paper, we introduce Text-Only Pre-Alignment (TOPA), a novel approach to extend large language models (LLMs) for video understanding, without the need for pre-training on real video data. Specifically, we first employ an advanced LLM to automatically generate Textual Videos comprising continuous textual frames, along with corresponding annotations to simulate real video-text data. Then, these annotated textual videos are used to pre-align a language-only LLM with the video modality. To bridge the gap between textual and real videos, we employ the CLIP model as the feature extractor to align image and text modalities. During text-only pre-alignment, the continuous textual frames, encoded as a sequence of CLIP text features, are analogous to continuous CLIP image features, thus aligning the LLM with real video representation. Extensive experiments, including zero-shot evaluation and finetuning on various video understanding tasks, demonstrate that TOPA is an effective and efficient framework for aligning video content with LLMs. In particular, without training on any video data, the TOPA-Llama2-13B model achieves a Top-1 accuracy of 51.0% on the challenging long-form video understanding benchmark, Egoschema. This performance surpasses previous video-text pre-training approaches and proves competitive with recent GPT-3.5-based video agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
