Shotluck Holmes：一系列用于视频字幕生成与摘要的高效小规模大型语言视觉模型
Shotluck Holmes: A Family of Efficient Small-Scale Large Language Vision Models For Video Captioning and Summarization
AI总结：
本文提出高效小型 LLVM 系列 Shotluck Holmes，通过改进预训练与数据策略使其理解帧序列，并在 Shot2Story 上以更小、更高效的模型取得领先性能。
摘要：
视频是一种日益重要且信息密集的媒介，但它也给语言模型带来了巨大挑战。典型视频由一系列较短的片段（即镜头）组成，这些片段共同构成连贯的叙事。每个镜头类似于句子中的一个词，其中必须同时处理多种信息流（例如视觉和听觉数据）。理解整个视频不仅需要理解每个镜头的视听信息，还要求模型将各个镜头之间的含义联系起来，以生成更宏大、涵盖全部内容的故事。尽管该领域已取得显著进展，但现有工作往往忽视视频中更细粒度的逐镜头语义信息。在本项目中，我们提出一系列高效的大型语言视觉模型（LLVM），以提升视频摘要和字幕生成能力，称为 Shotluck Holmes。通过利用更优的预训练和数据收集策略，我们将现有小型 LLVM 的能力从能够理解单张图片扩展到能够理解帧序列。具体而言，我们表明 Shotluck Holmes 在 Shot2Story 视频字幕生成和摘要任务上取得了优于当前最先进结果的性能，同时模型显著更小且计算效率更高。
英文摘要：
Video is an increasingly prominent and information-dense medium, yet it poses substantial challenges for language models. A typical video consists of a sequence of shorter segments, or shots, that collectively form a coherent narrative. Each shot is analogous to a word in a sentence where multiple data streams of information (such as visual and auditory data) must be processed simultaneously. Comprehension of the entire video requires not only understanding the visual-audio information of each shot but also requires that the model links the ideas between each shot to generate a larger, all-encompassing story. Despite significant progress in the field, current works often overlook videos' more granular shot-by-shot semantic information. In this project, we propose a family of efficient large language vision models (LLVMs) to boost video summarization and captioning called Shotluck Holmes. By leveraging better pretraining and data collection strategies, we extend the abilities of existing small LLVMs from being able to understand a picture to being able to understand a sequence of frames. Specifically, we show that Shotluck Holmes achieves better performance than state-of-the-art results on the Shot2Story video captioning and summary task with significantly smaller and more computationally efficient models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
