FreeVA：离线多模态大语言模型作为免训练视频助手
FreeVA: Offline MLLM as Training-Free Video Assistant
AI总结：
FreeVA 提出以免训练方式将离线图像多模态大语言模型直接用于视频问答，发现其零样本性能可超越视频指令微调方法，并指出 VideoInstruct-100K 微调未必带来增益及 GPT API 版本变化影响评估公平性。
摘要：
本文开展了一项实证研究，重新审视多模态大语言模型（MLLMs）的最新进展：视频助手。这项名为 FreeVA 的研究旨在以免训练的方式将现有的基于图像的 MLLM 扩展到视频领域。该研究提供了一个基础但必须了解的基线，并揭示了几个令人惊讶的发现：1）FreeVA 仅利用离线的基于图像的 MLLM 而无需额外训练，在零样本视频问答（例如 MSVD-QA、ActivityNet-QA 和 MSRVTT-QA）中表现出色，甚至超越了涉及视频指令微调的最先进方法。2）虽然主流的基于视频的 MLLM 通常以基于图像的 MLLM（例如 LLaVA）初始化，然后使用视频指令微调进行微调，但研究表明，利用广泛采用的 VideoInstruct-100K 进行视频指令微调实际上并不会比完全不训练带来更好的性能。3）现有工作中常用的评估指标会随着时间推移受到 GPT API 版本变化的显著影响。如果忽略这一点，可能会影响不同方法之间比较的公平性和一致性，并影响该领域研究人员的分析和判断。MLLM 的发展目前正蓬勃发展，吸引了众多研究人员进入该领域。我们希望这项工作能够作为一个即插即用、简单而有效的基线，鼓励在视频领域直接评估现有 MLLM，同时在一定程度上规范视频对话模型领域。此外，我们鼓励研究人员重新思考：当前的视频 MLLM 方法是否真正获得了超越图像 MLLM 的知识？代码可在 https://github.com/whwu95/FreeVA 获取。
英文摘要：
This paper undertakes an empirical study to revisit the latest advancements in Multimodal Large Language Models (MLLMs): Video Assistant. This study, namely FreeVA, aims to extend existing image-based MLLM to the video domain in a training-free manner. The study provides an essential, yet must-know baseline, and reveals several surprising findings: 1) FreeVA, leveraging only offline image-based MLLM without additional training, excels in zero-shot video question-answering (e.g., MSVD-QA, ActivityNet-QA, and MSRVTT-QA), even surpassing state-of-the-art methods that involve video instruction tuning. 2) While mainstream video-based MLLMs typically initialize with an image-based MLLM (e.g., LLaVA) and then fine-tune using video instruction tuning, the study indicates that utilizing the widely adopted VideoInstruct-100K for video instruction tuning doesn't actually lead to better performance compared to not training at all. 3) The commonly used evaluation metrics in existing works are significantly influenced by changes in the GPT API version over time. If ignored, this could affect the fairness and uniformity of comparisons between different methods and impact the analysis and judgment of researchers in the field. The advancement of MLLMs is currently thriving, drawing numerous researchers into the field. We aim for this work to serve as a plug-and-play, simple yet effective baseline, encouraging the direct evaluation of existing MLLMs in video domain while also standardizing the field of video conversational models to a certain extent. Also, we encourage researchers to reconsider: Have current video MLLM methods truly acquired knowledge beyond image MLLM? Code is available at https://github.com/whwu95/FreeVA
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
