AuroraCap：高效、高性能的视频详细描述生成与新基准
AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark
AI总结：
本文提出基于大型多模态模型的视频描述器AuroraCap，通过token合并策略实现高效视频详细描述，并构建VDC基准及LLM辅助评估指标VDCscore，在多个基准上超越GPT-4V等模型。
摘要：
视频详细描述生成是一项关键任务，旨在为视频内容生成全面且连贯的文本描述，对视频理解与生成均有裨益。本文提出AuroraCap，一个基于大型多模态模型的视频描述器。我们采用最简单的架构设计，不引入额外参数进行时序建模。为解决长视频序列带来的开销问题，我们实现了token合并策略，以减少输入视觉token的数量。令人意外的是，我们发现该策略几乎不造成性能损失。AuroraCap在多个视频和图像描述基准上展现出卓越性能，例如在Flickr30k上取得88.9的CIDEr分数，超过GPT-4V（55.3）和Gemini-1.5 Pro（82.2）。然而，现有视频描述基准仅包含由几十个词组成的简单描述，限制了该领域的研究。因此，我们构建了VDC，一个包含一千余条精心标注的结构化描述的视频详细描述基准。此外，我们提出了一种新的LLM辅助评估指标VDCscore以改进评估，该指标采用分而治之策略，将长描述评估转化为多个短问答对。借助人类Elo排名，我们的实验表明该基准与人类对视频详细描述质量的判断具有更好的相关性。
英文摘要：
Video detailed captioning is a key task which aims to generate comprehensive and coherent textual descriptions of video content, benefiting both video understanding and generation. In this paper, we propose AuroraCap, a video captioner based on a large multimodal model. We follow the simplest architecture design without additional parameters for temporal modeling. To address the overhead caused by lengthy video sequences, we implement the token merging strategy, reducing the number of input visual tokens. Surprisingly, we found that this strategy results in little performance loss. AuroraCap shows superior performance on various video and image captioning benchmarks, for example, obtaining a CIDEr of 88.9 on Flickr30k, beating GPT-4V (55.3) and Gemini-1.5 Pro (82.2). However, existing video caption benchmarks only include simple descriptions, consisting of a few dozen words, which limits research in this field. Therefore, we develop VDC, a video detailed captioning benchmark with over one thousand carefully annotated structured captions. In addition, we propose a new LLM-assisted metric VDCscore for bettering evaluation, which adopts a divide-and-conquer strategy to transform long caption evaluation into multiple short question-answer pairs. With the help of human Elo ranking, our experiments show that this benchmark better correlates with human judgments of video detailed captioning quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
