arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2503.04058cs.CV
收藏

EVE:基于视觉-语言模型的端到端视频字幕提取

EVE: Towards End-to-End Video Subtitle Extraction with Vision-Language Models

College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);ByteDance Inc.(字节跳动公司);College of Electronic and Information Engineering, Tongji University(同济大学电子与信息工程学院);School of Computing and Communications, Lancaster University(兰卡斯特大学计算机与通讯学院)

Haiyang Yu, Mengyang Zhao, Jinghui Lu, Ke Niu, Yanjie Wang, Weijie Yin, Weitao Jia, Teng Fu, Yang Liu, Jun Liu, Hong Chen

AI总结:

EVE提出了一种基于视觉-语言模型的端到端视频字幕提取框架,通过双分支模块高效提取字幕及时间戳,并构建了首个大规模视频字幕数据集。

摘要:

视频字幕在短视频和电影中起着关键作用,不仅帮助模型更好地理解视频内容,还支持视频翻译和内容检索等应用。现有的视频字幕提取方法通常依赖于多阶段框架,其中阶段间的误差累积以及由于逐帧处理而未能充分利用时间依赖性。此外,尽管一些大型视觉-语言模型(LVLMs)具有强大的OCR能力,但准确预测字幕文本的时间戳仍具有挑战性。为此,我们提出了一种基于LVLMs的端到端视频字幕提取框架,命名为EVE,可以同时输出字幕及其时间戳。具体而言,我们引入了一个双分支的空间时间字幕显著(S³)模块,作为LVLMs的适配器,能够表示与字幕相关的内容,并仅使用少量令牌考虑帧间相关性。在该模块中,空间语义上下文聚合分支聚合高级全局语义,以提供空间视觉上下文信息,而时间字幕令牌查询分支明确查询与字幕相关的令牌,同时考虑跨帧的时间相关性。由S³模块保留的少量令牌被馈送到语言模型,该模型随后直接输出字幕文本及其时间戳。此外,我们构建了首个大规模的视频字幕提取数据集ViSa,包含超过250万部带有时间戳和双语标注的视频,从而为社区提供了良好的训练和评估基准。

英文摘要:

Video subtitles play a crucial role in short videos and movies, as they not only help models better understand video content but also support applications such as video translation and content retrieval. Existing video subtitle extraction methods typically rely on multi-stage frameworks, where errors accumulate across stages and temporal dependencies are underutilized due to frame-wise processing. Moreover, although some Large Vision-Language Models (LVLMs) possess strong OCR capabilities, predicting accurate timestamps for subtitle texts remains challenging. To this end, we propose an End-to-end Video subtitle Extraction framework based on LVLMs, named EVE, which can output subtitles and their timestamps simultaneously. Specifically, we introduce a dual-branch Spatiotemporal Subtitle-Salient (S\textsuperscript{3}) Module that serves as an adapter for LVLMs, capable of representing subtitle-related content and considering inter-frame correlations using only a small number of tokens. Within this module, the Spatial Semantic Context Aggregate branch aggregates high-level global semantics to provide spatial visual contextual information, while the Temporal Subtitle Token Query branch explicitly queries subtitle-relevant tokens while considering temporal correlation across frames. The small number of tokens retained by the S\textsuperscript{3} module are fed to the language model, which then directly outputs the subtitle text along with its timestamps. Furthermore, we construct the first large-scale dataset dedicated to video subtitle extraction, ViSa, containing over 2.5M videos with timestamped and bilingual annotation, thereby providing the community with a well-organized training and evaluation benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑