InstanceCap：通过实例感知结构化字幕改进文本到视频生成
InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption
AI总结：
针对现有视频字幕细节不足、存在幻觉等影响文本到视频生成质量的问题，研究提出实例感知结构化字幕框架InstanceCap，构建InstanceVid数据集，实验证明其性能优于以往模型，提升了字幕与视频的保真度并减少幻觉。
摘要：
近年来，文本到视频生成发展迅速，取得了显著成果。其训练通常依赖视频-字幕配对数据，这类数据对提升生成性能起着关键作用。然而，当前的视频字幕常存在细节不足、幻觉和运动描述不精确的问题，影响了生成视频的保真度和一致性。在本研究中，我们提出了一种名为InstanceCap的新型实例感知结构化字幕框架，首次实现了实例级的细粒度视频字幕生成。基于该方案，我们设计了一个辅助模型集群，将原始视频转换为实例以提升实例保真度。我们进一步利用视频实例将密集提示优化为结构化短语，实现简洁而精确的描述。此外，我们构建了一个包含2.2万条数据的InstanceVid数据集用于训练，并提出了一个适配InstanceCap结构的推理增强流水线。实验结果表明，我们提出的InstanceCap显著优于以往模型，在减少幻觉的同时确保了字幕与视频之间的高保真度。
英文摘要：
Text-to-video generation has evolved rapidly in recent years, delivering remarkable results. Training typically relies on video-caption paired data, which plays a crucial role in enhancing generation performance. However, current video captions often suffer from insufficient details, hallucinations and imprecise motion depiction, affecting the fidelity and consistency of generated videos. In this work, we propose a novel instance-aware structured caption framework, termed InstanceCap, to achieve instance-level and fine-grained video caption for the first time. Based on this scheme, we design an auxiliary models cluster to convert original video into instances to enhance instance fidelity. Video instances are further used to refine dense prompts into structured phrases, achieving concise yet precise descriptions. Furthermore, a 22K InstanceVid dataset is curated for training, and an enhancement pipeline that tailored to InstanceCap structure is proposed for inference. Experimental results demonstrate that our proposed InstanceCap significantly outperform previous models, ensuring high fidelity between captions and videos while reducing hallucinations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
