Video DataFlywheel：解决视频-语言理解中不可能的数据三元悖论
Video DataFlywheel: Resolving the Impossible Data Trinity in Video-Language Understanding
AI总结：
本文提出Video DataFlywheel框架，通过迭代精炼视频标注并结合噪声控制方法AdaTaiLr，在保持多样性的同时提升数据质量，实现视频-语言理解性能提升3%。
摘要：
近年来，视频-语言理解通过大规模预训练取得了巨大成功。然而，数据稀缺仍然是一个普遍存在的挑战。本研究定量揭示了预训练数据集中数据数量、多样性和质量之间的“不可能三角”。近期的工作试图通过合成标注来精炼那些因低质量而受损的大规模、多样化ASR数据集。这些方法成功地利用了多模态视频内容（帧、标签、ASR转录文本等）中的有用信息来精炼原始标注。然而，它们难以缓解合成标注中的噪声，并且随着数据集规模的扩大而缺乏可扩展性。为了解决这些问题，我们提出了Video DataFlywheel框架，该框架通过改进的噪声控制方法迭代地精炼视频标注。在迭代精炼方面，我们首先利用视频-语言模型生成合成标注，从而得到一个精炼后的数据集。然后，我们在该数据集上进行预训练，并在人工精炼样本上进行微调，以获得更强的模型。这些过程被重复执行以实现持续改进。在噪声控制方面，我们提出了AdaTaiLr，这是一种新颖的噪声控制方法，它对噪声分布的要求更弱，因此在大规模数据集中被证明更为有效，并具有理论保证。迭代精炼与AdaTaiLr的结合能够在视频-语言理解中实现更好的可扩展性。大量实验表明，我们的框架优于现有的数据精炼基线方法，带来了3%的性能提升，并在最小化多样性损失的同时提高了数据集质量。此外，我们精炼后的数据集促进了各种视频-语言理解任务的显著改进，包括视频问答和文本-视频检索。
英文摘要：
Recently, video-language understanding has achieved great success through large-scale pre-training. However, data scarcity remains a prevailing challenge. This study quantitatively reveals an "impossible trinity" among data quantity, diversity, and quality in pre-training datasets. Recent efforts seek to refine large-scale, diverse ASR datasets compromised by low quality through synthetic annotations. These methods successfully leverage useful information in multimodal video content (frames, tags, ASR transcripts, etc.) to refine the original annotations. Nevertheless, they struggle to mitigate noise within synthetic annotations and lack scalability as the dataset size expands. To address these issues, we introduce the Video DataFlywheel framework, which iteratively refines video annotations with improved noise control methods. For iterative refinement, we first leverage a video-language model to generate synthetic annotations, resulting in a refined dataset. Then, we pre-train on it and fine-tune on human refinement examples for a stronger model. These processes are repeated for continuous improvement. For noise control, we present AdaTaiLr, a novel noise control method that requires weaker assumptions on noise distribution, thereby proving more effective in large datasets with theoretical guarantees. The combination of iterative refinement and AdaTaiLr can achieve better scalability in video-language understanding. Extensive experiments show that our framework outperforms existing data refinement baselines, delivering a 3% performance boost and improving dataset quality with minimal diversity loss. Furthermore, our refined dataset facilitates significant improvements in various video-language understanding tasks, including video question answering and text-video retrieval.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
