arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2411.12293cs.CVcs.HCcs.MM
收藏

面向指令式视觉装配的生成式时间线

Generative Timelines for Instructed Visual Assembly

Czech Institute of Informatics, Robotics and Cybernetics at the Czech Technical University in Prague(布拉格捷克技术大学信息学、机器人学与控制论研究所);AI Initiative, KAUST(阿卜杜拉国王科技大学AI计划);Adobe Research(奥多比研究院);CIIRC CTU(捷克技术大学信息学、机器人学与控制论研究所)

Alejandro Pardo, Jui-Hsien Wang, Bernard Ghanem, Josef Sivic, Bryan Russell, Fabian Caba Heilbron

AI总结:

本文提出Timeline Assembler生成式模型,通过大型多模态语言模型和自动数据集生成方法,实现基于自然语言指令的图像与视频时间线装配,并在复杂装配任务中显著优于GPT-4o等基线模型。

摘要:

本工作的目标是通过自然语言指令操纵视觉时间线(例如视频),使复杂的时间线编辑任务能够被非专家用户甚至可能是残障用户所使用。我们将这一任务称为指令式视觉装配(Instructed visual assembly)。该任务具有挑战性,因为它需要:(i)识别输入时间线中的相关视觉内容,并在给定的输入(视频)集合中检索相关视觉内容;(ii)理解输入的自然语言指令;以及(iii)对输入视觉时间线执行所需的编辑以生成输出时间线。为应对这些挑战,我们提出了Timeline Assembler,这是一个经过训练以执行指令式视觉装配任务的生成式模型。本工作的贡献有三方面。第一,我们开发了一个大型多模态语言模型,该模型旨在处理视觉内容、紧凑地表示时间线并准确解释时间线编辑指令。第二,我们引入了一种自动生成视觉装配任务数据集的新方法,从而无需人工标注数据即可高效训练我们的模型。第三,我们通过创建两个用于图像和视频装配的新数据集来验证我们的方法,证明Timeline Assembler在准确执行各种受真实世界启发的场景中的复杂装配指令方面,显著优于现有的基线模型,包括最近的GPT-4o。

英文摘要:

The objective of this work is to manipulate visual timelines (e.g. a video) through natural language instructions, making complex timeline editing tasks accessible to non-expert or potentially even disabled users. We call this task Instructed visual assembly. This task is challenging as it requires (i) identifying relevant visual content in the input timeline as well as retrieving relevant visual content in a given input (video) collection, (ii) understanding the input natural language instruction, and (iii) performing the desired edits of the input visual timeline to produce an output timeline. To address these challenges, we propose the Timeline Assembler, a generative model trained to perform instructed visual assembly tasks. The contributions of this work are three-fold. First, we develop a large multimodal language model, which is designed to process visual content, compactly represent timelines and accurately interpret timeline editing instructions. Second, we introduce a novel method for automatically generating datasets for visual assembly tasks, enabling efficient training of our model without the need for human-labeled data. Third, we validate our approach by creating two novel datasets for image and video assembly, demonstrating that the Timeline Assembler substantially outperforms established baseline models, including the recent GPT-4o, in accurately executing complex assembly instructions across various real-world inspired scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑