OmniVid：面向通用视频理解的生成式框架
OmniVid: A Generative Framework for Universal Video Understanding
AI总结：
OmniVid提出用语言、时间标记和边界框标记统一视频理解任务的输出空间，在共享的生成式编码器-解码器架构中处理分类、字幕生成和定位任务，并在七个基准上取得领先结果。
摘要：
视频理解任务（如识别、字幕生成和跟踪）的核心是自动检测视频中的物体或动作，并分析其随时间的演变。尽管这些任务共享共同目标，但不同任务通常依赖不同的模型架构和标注格式。相比之下，自然语言处理受益于统一的输出空间，即文本序列，这简化了以大量训练语料训练强大基础语言模型（如GPT-3）的过程。受此启发，我们试图通过使用语言作为标签，并额外引入时间标记和边界框标记，来统一视频理解任务的输出空间。通过这种方式，各种视频任务可以被形式化为基于视频的标记生成。这使我们能够在完全共享的编码器-解码器架构中，遵循生成式框架来处理各种类型的视频任务，包括分类（如动作识别）、字幕生成（涵盖片段字幕生成、视频问答和密集视频字幕生成）以及定位任务（如视觉目标跟踪）。通过全面的实验，我们证明了这种简单而直接的想法相当有效，能够在七个视频基准上取得最先进或具有竞争力的结果，为更通用的视频理解提供了新的视角。代码可在 https://github.com/wangjk666/OmniVid 获取。
英文摘要：
The core of video understanding tasks, such as recognition, captioning, and tracking, is to automatically detect objects or actions in a video and analyze their temporal evolution. Despite sharing a common goal, different tasks often rely on distinct model architectures and annotation formats. In contrast, natural language processing benefits from a unified output space, i.e., text sequences, which simplifies the training of powerful foundational language models, such as GPT-3, with extensive training corpora. Inspired by this, we seek to unify the output space of video understanding tasks by using languages as labels and additionally introducing time and box tokens. In this way, a variety of video tasks could be formulated as video-grounded token generation. This enables us to address various types of video tasks, including classification (such as action recognition), captioning (covering clip captioning, video question answering, and dense video captioning), and localization tasks (such as visual object tracking) within a fully shared encoder-decoder architecture, following a generative framework. Through comprehensive experiments, we demonstrate such a simple and straightforward idea is quite effective and can achieve state-of-the-art or competitive results on seven video benchmarks, providing a novel perspective for more universal video understanding. Code is available at https://github.com/wangjk666/OmniVid.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
