arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.15772cs.CVcs.AI
收藏

理解与生成:多模态模型中的优化困境导航

Understanding vs. Generation: Navigating Optimization Dilemma in Multimodal Models

State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室);Tencent(腾讯);Center for Data Science, Peking University(北京大学数据科学中心);Center for Machine Learning Research, Peking University(北京大学机器学习研究中心)

Sen Ye, Mengde Xu, Shuyang Gu, Di He, Liwei Wang, Han Hu

AI总结:

本文提出R3框架,通过将生成任务分解为生成-理解-生成的多步骤过程,缓解生成与理解之间的优化困境,提升生成效果和理解能力。

摘要:

当前多模态模型研究面临关键挑战:增强生成能力往往牺牲理解能力,反之亦然。我们分析了这一权衡,并发现生成与理解之间的潜在冲突可能是主要原因。为此,我们提出Reason-Reflect-Refine(R3)框架。该创新算法将单步生成任务重新定义为多步过程,即“生成-理解-生成”。通过在生成过程中显式利用模型的理解能力,我们成功缓解了优化困境,实现了更强的生成结果和改进的理解能力,这些能力与生成过程相关。这为设计下一代统一多模态模型提供了有价值的见解。代码可在https://github.com/sen-ye/R3获取。

英文摘要:

Current research in multimodal models faces a key challenge where enhancing generative capabilities often comes at the expense of understanding, and vice versa. We analyzed this trade-off and identify the primary cause might be the potential conflict between generation and understanding, which creates a competitive dynamic within the model. To address this, we propose the Reason-Reflect-Refine (R3) framework. This innovative algorithm re-frames the single-step generation task into a multi-step process of "generate-understand-regenerate". By explicitly leveraging the model's understanding capability during generation, we successfully mitigate the optimization dilemma, achieved stronger generation results and improved understanding ability which are related to the generation process. This offers valuable insights for designing next-generation unified multimodal models. Code is available at https://github.com/sen-ye/R3.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑