多句视频定位用于长视频生成
Multi-sentence Video Grounding for Long Video Generation
AI总结：
本文提出多句视频定位用于长视频生成的新范式，通过视频检索、编辑和变形，实现低内存成本且保持时间一致性的长视频生成。
摘要：
近年来，视频生成取得了巨大成功，但在生成长视频方面的应用仍然具有挑战性，这主要是由于难以保持生成视频的时间一致性以及生成过程中的高内存成本。为了解决这些问题，本文提出了一种新颖且大胆的思路——多句视频定位用于长视频生成，首次将大规模视频片段检索与视频生成任务联系起来，为长视频生成提供了新的范式。我们的工作方法可概括为三个步骤：（i）我们设计顺序场景文本提示作为视频定位的查询，利用大规模视频片段检索在视频数据库中搜索满足文本要求的视频片段。（ii）基于检索到的视频片段的源帧，我们采用视频编辑方法创建新的视频内容，同时保持检索视频的时间一致性。由于编辑可以逐段甚至逐帧进行，这大大降低了内存成本。（iii）我们还尝试了视频变形和个性化生成方法，以提高长视频生成的主体一致性，并为长视频生成的子任务提供了消融实验结果。我们的方法无缝地将图像/视频编辑、视频变形、个性化生成以及视频定位的发展扩展到长视频生成领域，为以低内存成本生成长视频提供了有效的解决方案。
英文摘要：
Video generation has witnessed great success recently, but their application in generating long videos still remains challenging due to the difficulty in maintaining the temporal consistency of generated videos and the high memory cost during generation. To tackle the problems, in this paper, we propose a brave and new idea of Multi-sentence Video Grounding for Long Video Generation, connecting the massive video moment retrieval to the video generation task for the first time, providing a new paradigm for long video generation. The method of our work can be summarized as three steps: (i) We design sequential scene text prompts as the queries for video grounding, utilizing the massive video moment retrieval to search for video moment segments that meet the text requirements in the video database. (ii) Based on the source frames of retrieved video moment segments, we adopt video editing methods to create new video content while preserving the temporal consistency of the retrieved video. Since the editing can be conducted segment by segment, and even frame by frame, it largely reduces the memory cost. (iii) We also attempt video morphing and personalized generation methods to improve the subject consistency of long video generation, providing ablation experimental results for the subtasks of long video generation. Our approach seamlessly extends the development in image/video editing, video morphing and personalized generation, and video grounding to the long video generation, offering effective solutions for generating long videos at low memory cost.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
