VCoME:具有多模态编辑效果的口播视频合成
VCoME: Verbal Video Composition with Multimodal Editing Effects
AI总结:
本文提出带多模态编辑效果的口播视频合成任务及大规模数据集,并构建基于大型多模态模型的 VCoME 框架,自动推荐效果位置与类型,效率达专业编辑的85倍。
摘要:
以旁白或文字叠加为特征的口播视频提供了有价值的内容,但在合成方面面临重大挑战,尤其是在融入编辑效果以提升清晰度和视觉吸引力时。本文提出了一项新颖的任务,即带编辑效果的口播视频合成。该任务旨在通过整合文本、视觉和音频类别的多模态编辑效果,生成连贯且具有视觉吸引力的口播视频。为实现这一目标,我们从公开来源整理了一个大规模视频效果合成数据集。随后,我们将该任务构建为一个生成问题,包括识别口播内容中的合适位置,以及为这些位置推荐编辑效果。为解决该任务,我们提出了 VCoME,这是一个采用大型多模态模型为视频合成生成编辑效果的通用框架。具体而言,VCoME 接收多模态视频上下文,并自回归地输出在口播内容中何处应用效果,以及每个位置最适合哪些效果。VCoME 还支持基于提示的合成密度和风格控制,为多样化应用提供了充分的灵活性。通过广泛的定量和定性评估,我们清楚地证明了 VCoME 的有效性。一项全面的用户研究表明,我们的方法能够生成具有专业质量的视频,同时效率比专业编辑人员高 85 倍。
英文摘要:
Verbal videos, featuring voice-overs or text overlays, provide valuable content but present significant challenges in composition, especially when incorporating editing effects to enhance clarity and visual appeal. In this paper, we introduce the novel task of verbal video composition with editing effects. This task aims to generate coherent and visually appealing verbal videos by integrating multimodal editing effects across textual, visual, and audio categories. To achieve this, we curate a large-scale dataset of video effects compositions from publicly available sources. We then formulate this task as a generative problem, involving the identification of appropriate positions in the verbal content and the recommendation of editing effects for these positions. To address this task, we propose VCoME, a general framework that employs a large multimodal model to generate editing effects for video composition. Specifically, VCoME takes in the multimodal video context and autoregressively outputs where to apply effects within the verbal content and which effects are most appropriate for each position. VCoME also supports prompt-based control of composition density and style, providing substantial flexibility for diverse applications. Through extensive quantitative and qualitative evaluations, we clearly demonstrate the effectiveness of VCoME. A comprehensive user study shows that our method produces videos of professional quality while being 85$\times$ more efficient than professional editors.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。