arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.10484cs.CV
收藏

超越原始视频:利用大型多模态模型理解编辑视频

Beyond Raw Videos: Understanding Edited Videos with Large Multimodal Model

ByteDance Inc.(字节跳动公司)

Lu Xu, Sijie Zhu, Chunyuan Li, Chia-Wen Kuo, Fan Chen, Xinyao Wang, Guang Chen, Dawei Du, Ye Yuan, Longyin Wen

AI总结:

本文构建了基于TikTok编辑视频的EditVid-QA基准,揭示现有视频LMM在社交媒体编辑视频上的领域差距,并通过混合原始与编辑视频的训练数据提升模型泛化能力,同时指出并规避了GPT-3.5评估中的“抱歉”攻击问题。

摘要:

新兴的视频大型多模态模型(LMM)在以视觉问答(VQA)形式进行的通用视频理解方面取得了显著进展,其中原始视频由摄像机拍摄。然而,现实世界应用中的很大一部分视频是编辑视频,例如,用户通常在将原始视频发布到社交媒体平台之前对其进行剪辑并添加特效/修改。这些编辑视频通常具有很高的观看量,但现有的视频LMM基准(如ActivityNet-QA或VideoChatGPT基准)并未覆盖它们。本文利用热门短视频平台TikTok上的编辑视频,构建了一个视频VQA基准(命名为EditVid-QA),涵盖四类典型编辑类别,即特效、搞笑、梗图和游戏。搞笑和梗图视频基准测试细致入微的理解和高层次推理能力,而特效和游戏则评估对人工设计的理解能力。大多数开源视频LMM在EditVid-QA基准上表现不佳,表明社交媒体上的编辑短视频与常规原始视频之间存在巨大的领域差距。为了提高LMM的泛化能力,我们基于Panda-70M/WebVid原始视频和小规模TikTok/CapCut编辑视频,为所提出的基准收集了一个训练集,这提升了在所提出的EditVid-QA基准上的性能,表明高质量训练数据的有效性。我们还发现了使用GPT-3.5评判器的现有评估协议中的一个严重问题,即“抱歉”攻击,在这种攻击中,一种抱歉式的朴素回答可以从GPT评判器获得极高的评分,例如在VideoChatGPT评估协议上的正确性得分超过4.3。为避免“抱歉”攻击,我们使用GPT-4评判器和关键词过滤来评估结果。数据集已在https://github.com/XenonLamb/EditVid-QA发布。

英文摘要:

The emerging video LMMs (Large Multimodal Models) have achieved significant improvements on generic video understanding in the form of VQA (Visual Question Answering), where the raw videos are captured by cameras. However, a large portion of videos in real-world applications are edited videos, \textit{e.g.}, users usually cut and add effects/modifications to the raw video before publishing it on social media platforms. The edited videos usually have high view counts but they are not covered in existing benchmarks of video LMMs, \textit{i.e.}, ActivityNet-QA, or VideoChatGPT benchmark. In this paper, we leverage the edited videos on a popular short video platform, \textit{i.e.}, TikTok, and build a video VQA benchmark (named EditVid-QA) covering four typical editing categories, i.e., effect, funny, meme, and game. Funny and meme videos benchmark nuanced understanding and high-level reasoning, while effect and game evaluate the understanding capability of artificial design. Most of the open-source video LMMs perform poorly on the EditVid-QA benchmark, indicating a huge domain gap between edited short videos on social media and regular raw videos. To improve the generalization ability of LMMs, we collect a training set for the proposed benchmark based on both Panda-70M/WebVid raw videos and small-scale TikTok/CapCut edited videos, which boosts the performance on the proposed EditVid-QA benchmark, indicating the effectiveness of high-quality training data. We also identified a serious issue in the existing evaluation protocol using the GPT-3.5 judge, namely a "sorry" attack, where a sorry-style naive answer can achieve an extremely high rating from the GPT judge, e.g., over 4.3 for correctness score on VideoChatGPT evaluation protocol. To avoid the "sorry" attacks, we evaluate results with GPT-4 judge and keyword filtering. The dataset is released at https://github.com/XenonLamb/EditVid-QA.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑