arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.18543cs.CV
收藏

GenAgent:通过代理多模态推理扩展文本到图像生成

GenAgent: Scaling Text-to-Image Generation via Agentic Multimodal Reasoning

Fudan University(复旦大学);Tongyi Lab(通义实验室);Nanjing University(南京大学);University of Science and Technology of China(中国科学技术大学)

Kaixun Jiang, Yuzheng Wang, Junjie Zhou, Pandeng Li, Zhihang Liu, Chen-Wei Xie, Zhaoyu Chen, Yun Zheng, Wenqiang Zhang

AI总结:

GenAgent通过代理多模态推理提升文本到图像生成性能,采用两阶段训练策略实现自主多轮交互与任务自适应推理。

摘要:

我们介绍了GenAgent,通过一个代理多模态模型统一视觉理解和生成。与面临昂贵训练成本和理解-生成权衡的统一模型不同,GenAgent通过代理框架解耦这些能力:理解由多模态模型本身处理,而生成则通过将图像生成模型视为可调用的工具来实现。关键的是,与现有受静态流水线限制的模块化系统不同,这种设计使代理能够进行自主的多轮交互,其中代理生成涵盖推理、工具调用、判断和反思的多模态思维链,以迭代改进输出。我们采用两阶段训练策略:首先,通过高质量的工具调用和反思数据进行监督微调,以引导代理行为;其次,端到端的代理强化学习结合点奖励(最终图像质量)和成对奖励(反思准确性),并使用轨迹重采样以增强多轮探索。GenAgent显著提升了基生成器(FLUX.1-dev)在GenEval++(+23.6%)和WISE(+14%)上的性能。除了性能提升,我们的框架展示了三个关键特性:1)跨工具泛化到具有不同能力的生成器,2)测试时的扩展,随着交互轮次的一致改进,3)任务自适应推理,能够自动调整到不同任务。我们的代码将在https://github.com/deep-kaixun/GenAgent处提供。

英文摘要:

We introduce GenAgent, unifying visual understanding and generation through an agentic multimodal model. Unlike unified models that face expensive training costs and understanding-generation trade-offs, GenAgent decouples these capabilities through an agentic framework: understanding is handled by the multimodal model itself, while generation is achieved by treating image generation models as invokable tools. Crucially, unlike existing modular systems constrained by static pipelines, this design enables autonomous multi-turn interactions where the agent generates multimodal chains-of-thought encompassing reasoning, tool invocation, judgment, and reflection to iteratively refine outputs. We employ a two-stage training strategy: first, cold-start with supervised fine-tuning on high-quality tool invocation and reflection data to bootstrap agent behaviors; second, end-to-end agentic reinforcement learning combining pointwise rewards (final image quality) and pairwise rewards (reflection accuracy), with trajectory resampling for enhanced multi-turn exploration. GenAgent significantly boosts base generator(FLUX.1-dev) performance on GenEval++ (+23.6\%) and WISE (+14\%). Beyond performance gains, our framework demonstrates three key properties: 1) cross-tool generalization to generators with varying capabilities, 2) test-time scaling with consistent improvements across interaction rounds, and 3) task-adaptive reasoning that automatically adjusts to different tasks. Our code will be available at \href{https://github.com/deep-kaixun/GenAgent}{this url}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑