arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.05422cs.CV
收藏

ParaUni: 通过强化驱动的分层并行信息交互增强统一多模态模型的生成

ParaUni: Enhance Generation in Unified Multimodal Model with Reinforcement-driven Hierarchical Parallel Information Interaction

MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China(中国科学技术大学,教育部类脑智能感知与认知重点实验室);Huawei Inc.(华为技术有限公司)

Jiangtong Tan, Lin Liu, Jie Huanng, Xiaopeng Zhang, Qi Tian, Feng Zhao

AI总结:

ParaUni通过并行提取多层视觉语言模型特征并结合强化学习动态调整机制,提升统一多模态模型的生成能力。

摘要:

统一多模态模型通过结合视觉语言模型(VLMs)和扩散模型显著提升了视觉生成能力。然而,现有方法难以在充分交互与灵活实现之间取得平衡,由于表示差异巨大。考虑到VLM各层中从低级细节到高级语义的丰富层次信息,我们提出了ParaUni。它通过并行方式从不同VLM层提取特征以实现全面的信息交互,并保留灵活的分离架构以增强统一多模态模型的生成能力。具体而言,所有VLM层的视觉特征并行输入到层整合模块(LIM),高效整合细粒度细节和语义抽象,并将融合的表示作为条件输入扩散模型。为进一步提升性能,我们发现这些层次化层对不同强化学习(RL)奖励的响应不一致。关键在于,我们设计了层间动态调整机制(LDAM),以促进多种奖励改进,通过RL对这些层的层次特性进行对齐。大量实验表明,ParaUni利用互补的多层特征显著提升了生成质量,并在RL阶段展现出强大的多奖励提升潜力。代码可在https://github.com/JosephTiTan/ParaUni获取。

英文摘要:

Unified multimodal models significantly improve visual generation by combining vision-language models (VLMs) with diffusion models. However, existing methods struggle to fully balance sufficient interaction and flexible implementation due to vast representation difference. Considering abundant and hierarchical information in VLM's layers from low-level details to high-level semantics, we propose \textbf{ParaUni}. It extracts features from variants VLM's layers in a \textbf{Para}llel way for comprehensive information interaction and retains a flexible separation architecture to enhance generation in \textbf{Uni}fied multimodal model. Concretely, visual features from all VLM's layers are fed in parallel into a Layer Integration Module (LIM), which efficiently integrates fine-grained details and semantic abstractions and provides the fused representation as a condition to the diffusion model. To further enhance performance, we reveal that these hierarchical layers respond unequally to different rewards in Reinforcement Learning (RL). Crucially, we design a Layer-wise Dynamic Adjustment Mechanism (LDAM) to facilitate multiple reward improvements that aligns the hierarchical properties of these layers using RL. Extensive experiments show ParaUni leverages complementary multi-layer features to substantially improve generation quality and shows strong potential for multiple reward advances during RL stages. Code is available at https://github.com/JosephTiTan/ParaUni.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑