TWLV-I：视频基础模型全面评估的分析与见解
TWLV-I: Analysis and Insights from Holistic Evaluation on Video Foundation Models
AI总结：
针对视频基础模型评估参数不一致的问题，提出了一个公平稳健的评估框架，并引入了新的视频基础模型TWLV-I，在多个动作识别基准上显著优于现有模型。
摘要：
在这项工作中，我们讨论以公平且稳健的方式评估视频基础模型。与语言或图像基础模型不同，许多视频基础模型在评估时使用了不同的参数（如采样率、帧数、预训练步数等），这使得公平且稳健的比较具有挑战性。因此，我们提出了一个精心设计的评估框架，用于衡量视频理解的两个核心能力：外观和运动理解。我们的发现表明，现有的视频基础模型，无论是像UMT或InternVideo2这样的文本监督模型，还是像V-JEPA这样的自监督模型，在至少一项能力上表现出局限性。作为替代方案，我们引入了TWLV-I，这是一种新的视频基础模型，能够为基于运动和基于外观的视频构建稳健的视觉表示。基于在五个动作识别基准上linear probing的平均top-1准确率，仅在公开可访问的数据集上进行预训练，我们的模型相比V-JEPA (ViT-L)显示出4.6%p的提升，相比UMT (ViT-L)显示出7.7%p的提升。即使与更大规模的模型相比，我们的模型相比DFN (ViT-H)也展现出7.2%p的提升，相比V-JEPA (ViT-H)有2.7%p的提升，相比InternVideo2 (ViT-g)有2.8%p的提升。我们提供了TWLV-I从几个常用视频基准的视频中获得的embedding vectors，以及可以直接利用这些嵌入的评估源代码。代码可在https://github.com/twelvelabs-io/video-embeddings-evaluation-framework获取。
英文摘要：
In this work, we discuss evaluating video foundation models in a fair and robust manner. Unlike language or image foundation models, many video foundation models are evaluated with differing parameters (such as sampling rate, number of frames, pretraining steps, etc.), making fair and robust comparisons challenging. Therefore, we present a carefully designed evaluation framework for measuring two core capabilities of video comprehension: appearance and motion understanding. Our findings reveal that existing video foundation models, whether text-supervised like UMT or InternVideo2, or self-supervised like V-JEPA, exhibit limitations in at least one of these capabilities. As an alternative, we introduce TWLV-I, a new video foundation model that constructs robust visual representations for both motion- and appearance-based videos. Based on the average top-1 accuracy of linear probing on five action recognition benchmarks, pretrained only on publicly accessible datasets, our model shows a 4.6%p improvement compared to V-JEPA (ViT-L) and a 7.7%p improvement compared to UMT (ViT-L). Even when compared to much larger models, our model demonstrates a 7.2%p improvement compared to DFN (ViT-H), a 2.7%p improvement compared to V-JEPA (ViT-H) and a 2.8%p improvement compared to InternVideo2 (ViT-g). We provide embedding vectors obtained by TWLV-I from videos of several commonly used video benchmarks, along with evaluation source code that can directly utilize these embeddings. The code is available at https://github.com/twelvelabs-io/video-embeddings-evaluation-framework.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
