VC-LLM：基于多模态大语言模型从原始素材自动生成广告视频
VC-LLM: Automated Advertisement Video Creation from Raw Footage using Multi-modal LLMs
AI总结：
本文提出VC-LLM框架，利用多模态大语言模型结合高分辨率空间与低分辨率时间输入，并引入可追溯补充信息减少幻觉，实现从原始素材自动生成高质量短视频广告。
摘要：
随着短视频日益流行，视频内容在广告中的作用变得越来越重要。通常，广告主会围绕产品录制大量原始素材，然后基于这些原始素材制作大量不同的短视频广告。制作此类视频主要涉及编辑原始素材和撰写广告脚本，这需要一定的创意能力。为同一产品制作大量不同的视频内容通常具有挑战性，且人工效率往往较低。在本文中，我们提出了VC-LLM，一个由大语言模型驱动的框架，用于自动生成高质量的短视频广告。我们的方法利用高分辨率空间输入和低分辨率时间输入来更有效地表示视频片段，同时捕捉细粒度的视觉细节和更广泛的时间动态。此外，在训练过程中，我们引入通过改写真实文本生成的补充信息，确保所有关键输出信息都能直接追溯到输入，从而减少模型幻觉。我们还设计了一个基准来评估所生成视频的质量。实验表明，基于GPT-4o的VC-LLM能够生成与人工创作相当的视频。此外，我们收集了大量高质量短视频广告以构建预训练数据集，并人工清洗了部分数据以构建高质量微调数据集。实验表明，在该基准上，基于微调大语言模型的VC-LLM所生成的视频在叙事逻辑上优于基于GPT-4o的VC-LLM所生成的视频。
英文摘要：
As short videos have risen in popularity, the role of video content in advertising has become increasingly significant. Typically, advertisers record a large amount of raw footage about the product and then create numerous different short-form advertisement videos based on this raw footage. Creating such videos mainly involves editing raw footage and writing advertisement scripts, which requires a certain level of creative ability. It is usually challenging to create many different video contents for the same product, and manual efficiency is often low. In this paper, we present VC-LLM, a framework powered by Large Language Models for the automatic creation of high-quality short-form advertisement videos. Our approach leverages high-resolution spatial input and low-resolution temporal input to represent video clips more effectively, capturing both fine-grained visual details and broader temporal dynamics. In addition, during training, we incorporate supplementary information generated by rewriting the ground truth text, ensuring that all key output information can be directly traced back to the input, thereby reducing model hallucinations. We also designed a benchmark to evaluate the quality of the created videos. Experiments show that VC-LLM based on GPT-4o can produce videos comparable to those created by humans. Furthermore, we collected numerous high-quality short advertisement videos to create a pre-training dataset and manually cleaned a portion of the data to construct a high-quality fine-tuning dataset. Experiments indicate that, on the benchmark, the VC-LLM based on fine-tuned LLM can produce videos with superior narrative logic compared to those created by the VC-LLM based on GPT-4o.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
