REGen：面向长视频到短视频编辑的多模态检索-嵌入生成
REGen: Multimodal Retrieval-Embedded Generation for Long-to-Short Video Editing
AI总结：
REGen提出检索-嵌入生成框架，结合微调大模型与检索模型，在长视频中提取片段嵌入短视频，生成连贯叙事，并在纪录片预告片任务上优于现有方法。
摘要：
短视频是推广内容和提高知识可获取性的有效工具。现有的抽取式视频摘要方法难以生成连贯的叙事，而现有的生成式方法无法从输入视频中“引用”，即在输出中插入短视频片段。在这项工作中，我们探索新颖的视频编辑模型，用于生成具有连贯叙事并嵌入从长输入视频中提取的视频片段的短视频。我们提出了一种新颖的检索-嵌入生成框架，该框架允许大型语言模型在保持连贯叙事的同时引用多模态资源。我们提出的REGen系统首先使用微调的大型语言模型生成带有引用占位符的输出故事脚本，然后使用新颖的检索模型通过从候选可引用视频片段池中选择最能支持叙事的视频片段来替换引用占位符。我们在纪录片预告片生成任务上检验了所提出的方法，该任务中通常使用简短的采访插入来支持纪录片的叙事。我们的客观评估表明，所提出的方法能够有效地插入短视频片段，同时保持连贯的叙事。在一项主观调查中，我们表明，在预告片生成的连贯性、对齐性和真实性方面，我们提出的方法优于现有的生成式和抽取式方法。
英文摘要：
Short videos are an effective tool for promoting contents and improving knowledge accessibility. While existing extractive video summarization methods struggle to produce a coherent narrative, existing abstractive methods cannot `quote' from the input videos, i.e., inserting short video clips in their outputs. In this work, we explore novel video editing models for generating shorts that feature a coherent narrative with embedded video insertions extracted from a long input video. We propose a novel retrieval-embedded generation framework that allows a large language model to quote multimodal resources while maintaining a coherent narrative. Our proposed REGen system first generates the output story script with quote placeholders using a finetuned large language model, and then uses a novel retrieval model to replace the quote placeholders by selecting a video clip that best supports the narrative from a pool of candidate quotable video clips. We examine the proposed method on the task of documentary teaser generation, where short interview insertions are commonly used to support the narrative of a documentary. Our objective evaluations show that the proposed method can effectively insert short video clips while maintaining a coherent narrative. In a subjective survey, we show that our proposed method outperforms existing abstractive and extractive approaches in terms of coherence, alignment, and realism in teaser generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
