多功能多模态代理用于多媒体内容生成
A Versatile Multimodal Agent for Multimedia Content Generation
AI总结：
本文提出了一种多功能多模态代理，通过技能获取理论和两阶段相关策略，实现复杂多媒体内容生成任务的自动化。
摘要：
随着AIGC（人工智能生成内容）技术的发展，越来越多的生成模型正在革新视频编辑、音乐生成甚至电影制作等领域。然而，由于当前AIGC模型的限制，大多数模型只能作为特定应用场景中的单个组件，无法在现实应用中完成端到端的任务。在现实应用中，编辑专家往往需要处理各种图像和视频输入，生成多模态输出——视频通常包括音频、文本和其他元素。这种跨多种模态的整合目前的模型无法有效实现。然而，代理系统的发展使得使用AI工具解决复杂内容生成任务成为可能。为应对复杂场景，本文提出了一种MultiMedia-Agent，旨在自动化复杂内容创作。我们的代理系统包括数据生成管道、内容创作工具库和一组评估偏好对齐的指标。值得注意的是，我们引入了技能获取理论来建模训练数据的整理和代理训练。我们设计了一种两阶段相关策略用于计划优化，包括自相关和模型偏好相关。此外，我们利用生成的计划通过三阶段方法对MultiMedia-Agent进行训练，包括基础/成功计划微调和偏好优化。比较结果表明，我们的方法有效，MultiMedia-Agent能够生成比新型模型更好的多媒体内容。
英文摘要：
With the advancement of AIGC (AI-generated content) technologies, an increasing number of generative models are revolutionizing fields such as video editing, music generation, and even film production. However, due to the limitations of current AIGC models, most models can only serve as individual components within specific application scenarios and are not capable of completing tasks end-to-end in real-world applications. In real-world applications, editing experts often work with a wide variety of images and video inputs, producing multimodal outputs -- a video typically includes audio, text, and other elements. This level of integration across multiple modalities is something current models are unable to achieve effectively. However, the rise of agent-based systems has made it possible to use AI tools to tackle complex content generation tasks. To deal with the complex scenarios, in this paper, we propose a MultiMedia-Agent designed to automate complex content creation. Our agent system includes a data generation pipeline, a tool library for content creation, and a set of metrics for evaluating preference alignment. Notably, we introduce the skill acquisition theory to model the training data curation and agent training. We designed a two-stage correlation strategy for plan optimization, including self-correlation and model preference correlation. Additionally, we utilized the generated plans to train the MultiMedia-Agent via a three stage approach including base/success plan finetune and preference optimization. The comparison results demonstrate that the our approaches are effective and the MultiMedia-Agent can generate better multimedia content compared to novel models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
