VAST：一种视觉-音频-字幕-文本全模态基础模型及数据集
VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset
AI总结：
针对现有视频-文本基础模型对音频、字幕模态关注不足的问题，该研究构建了2700万样本的全模态视频字幕数据集VAST-27M，训练出能处理多模态的VAST基础模型，在22项跨模态基准上取得SOTA结果。
摘要：
在当前的视频-文本基础模型中，视觉与文本模态已得到充分研究，但视频中的音频、字幕等其他模态尚未获得足够关注。本文通过探索一个自动生成的大规模全模态视频字幕数据集VAST-27M，旨在建立视觉、音频、字幕等多模态视频轨道与文本之间的关联。具体而言，我们首先收集了2700万条开放域视频片段，并分别训练视觉字幕生成器与音频字幕生成器，以生成视觉和音频对应的字幕。随后，我们采用现成的大语言模型（LLM），将生成的字幕与字幕文本、指令提示词整合，生成全模态字幕。基于所提出的VAST-27M数据集，我们训练了名为VAST的全模态视频-文本基础模型，该模型能够感知和处理视频中的视觉、音频、字幕模态，更好地支持视觉-文本、音频-文本以及多模态视频-文本各类任务（包括检索、字幕生成和问答）。我们开展了大量实验以验证所提出的VAST-27M语料库与VAST基础模型的有效性，VAST在多个跨模态基准测试上取得了22项新的最优结果。代码、模型和数据集将发布于https://github.com/TXH-mercury/VAST。
英文摘要：
Vision and text have been fully explored in contemporary video-text foundational models, while other modalities such as audio and subtitles in videos have not received sufficient attention. In this paper, we resort to establish connections between multi-modality video tracks, including Vision, Audio, and Subtitle, and Text by exploring an automatically generated large-scale omni-modality video caption dataset called VAST-27M. Specifically, we first collect 27 million open-domain video clips and separately train a vision and an audio captioner to generate vision and audio captions. Then, we employ an off-the-shelf Large Language Model (LLM) to integrate the generated captions, together with subtitles and instructional prompts into omni-modality captions. Based on the proposed VAST-27M dataset, we train an omni-modality video-text foundational model named VAST, which can perceive and process vision, audio, and subtitle modalities from video, and better support various tasks including vision-text, audio-text, and multi-modal video-text tasks (retrieval, captioning and QA). Extensive experiments have been conducted to demonstrate the effectiveness of our proposed VAST-27M corpus and VAST foundation model. VAST achieves 22 new state-of-the-art results on various cross-modality benchmarks. Code, model and dataset will be released at https://github.com/TXH-mercury/VAST.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
