想象一下！从作曲到视频
Imagine This! Scripts to Compositions to Videos
摘要：
想象一个用自然语言描述的场景，具有实体的现实布局和外观，是对空间、视觉和语义世界知识的最终考验。为了实现这一目标，我们提出了组成，检索和融合网络（CRAFT），一个模型能够学习这些知识，从视频字幕数据和应用它，同时从新的字幕生成视频。CRAFT显式地预测所提到的实体（人物和对象）的时间布局，从视频数据库中检索时空实体片段并将它们融合以生成场景视频。我们的贡献包括CRAFT组件的顺序训练，同时联合建模布局和外观，以及鼓励学习组合表示检索的损失。我们评估CRAFT对标题的语义保真度，构图一致性和视觉质量。CRAFT优于直接像素生成方法，并且可以很好地推广到看不见的字幕和没有文本注释的看不见的视频数据库。我们展示了工艺上火石，一个新的丰富的注释视频字幕数据集超过25000个视频。有关CRAFT生成的视频的一瞥，请参见https://youtu.be/688Vv86n0z8。
英文摘要：
Imagining a scene described in natural language with realistic layout and appearance of entities is the ultimate test of spatial, visual, and semantic world knowledge. Towards this goal, we present the Composition, Retrieval, and Fusion Network (CRAFT), a model capable of learning this knowledge from video-caption data and applying it while generating videos from novel captions. CRAFT explicitly predicts a temporal-layout of mentioned entities (characters and objects), retrieves spatio-temporal entity segments from a video database and fuses them to generate scene videos. Our contributions include sequential training of components of CRAFT while jointly modeling layout and appearances, and losses that encourage learning compositional representations for retrieval. We evaluate CRAFT on semantic fidelity to caption, composition consistency, and visual quality. CRAFT outperforms direct pixel generation approaches and generalizes well to unseen captions and to unseen video databases with no text annotations. We demonstrate CRAFT on FLINTSTONES, a new richly annotated video-caption dataset with over 25000 videos. For a glimpse of videos generated by CRAFT, see https://youtu.be/688Vv86n0z8.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
