arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.17693cs.HCcs.AI
收藏

ExpressEdit:基于自然语言与草图的视频编辑

ExpressEdit: Video Editing with Natural Language and Sketching

KAIST(韩国科学技术院);Yale University(耶鲁大学)

Bekzat Tilekbay, Saelyne Yang, Michal Lewkowicz, Alex Suryapranata, Juho Kim

AI总结:

针对新手视频编辑困难的问题,提出ExpressEdit系统,利用自然语言和草图输入,结合LLM与视觉模型解析多模态编辑意图并支持迭代,显著提升了编辑效率与创意表达。

摘要:

信息类视频是向新手和专家解释概念性及程序性知识的重要来源。在制作信息类视频时,编辑者通过叠加文本/图像或裁剪片段来编辑视频,以提升视频质量并使其更具吸引力。然而,视频编辑可能既困难又耗时,尤其是对于新手视频编辑者而言,他们常常难以表达和实现其编辑想法。为了应对这一挑战,我们首先探索了如何利用多模态——自然语言(NL)和草图,这些是人类用于表达的自然模态——来支持视频编辑者表达视频编辑想法。我们从10位视频编辑者那里收集了176条编辑命令的多模态表达,揭示了NL和草图在描述编辑意图时的使用模式。基于这些发现,我们提出了ExpressEdit,这是一个通过NL文本和在视频帧上绘制草图来实现视频编辑的系统。在大型语言模型(LLM)和视觉模型的驱动下,该系统能够解释NL命令中的(1)时间、(2)空间和(3)操作引用,以及来自草图的空间引用。系统会执行这些被解析出的编辑操作,随后用户可以对其进行迭代。一项观察性研究(N=10)表明,ExpressEdit增强了新手视频编辑者表达和实现其编辑想法的能力。该系统允许参与者更高效地执行编辑操作,并通过基于用户的多模态编辑命令生成编辑结果以及支持对编辑命令的迭代,来激发更多想法。这项工作为未来多模态界面和基于AI的视频编辑流程设计提供了见解。

英文摘要:

Informational videos serve as a crucial source for explaining conceptual and procedural knowledge to novices and experts alike. When producing informational videos, editors edit videos by overlaying text/images or trimming footage to enhance the video quality and make it more engaging. However, video editing can be difficult and time-consuming, especially for novice video editors who often struggle with expressing and implementing their editing ideas. To address this challenge, we first explored how multimodality$-$natural language (NL) and sketching, which are natural modalities humans use for expression$-$can be utilized to support video editors in expressing video editing ideas. We gathered 176 multimodal expressions of editing commands from 10 video editors, which revealed the patterns of use of NL and sketching in describing edit intents. Based on the findings, we present ExpressEdit, a system that enables editing videos via NL text and sketching on the video frame. Powered by LLM and vision models, the system interprets (1) temporal, (2) spatial, and (3) operational references in an NL command and spatial references from sketching. The system implements the interpreted edits, which then the user can iterate on. An observational study (N=10) showed that ExpressEdit enhanced the ability of novice video editors to express and implement their edit ideas. The system allowed participants to perform edits more efficiently and generate more ideas by generating edits based on user's multimodal edit commands and supporting iterations on the editing commands. This work offers insights into the design of future multimodal interfaces and AI-based pipelines for video editing.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑