Uni-X: 通过双端分离架构缓解多模态冲突以实现统一多模态模型
Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Models
AI总结：
Uni-X通过双端分离和中间共享的架构缓解多模态冲突，提升统一多模态模型的训练效率和性能。
摘要：
统一多模态模型（UMMs）基于共享自回归（AR）变压器架构，因其架构简单而具有吸引力。然而，我们发现一个关键限制：当在多模态输入上进行训练时，模态共享的变压器会遭受视觉和文本之间严重的梯度冲突，尤其是在浅层和深层。我们将这一问题归因于图像和文本在低级统计特性上的根本差异，同时注意到在中间层中，表示变得更加抽象且语义对齐，冲突会有所减少。为克服这一挑战，我们提出了Uni-X，一种双端分离、中间共享的架构。Uni-X将初始和最终层专门用于模态特定处理，同时在中间层保持共享参数以实现高层语义融合。这种X型设计不仅消除了两端的梯度冲突，还进一步减轻了共享层中的残留冲突。广泛的实验验证了Uni-X的有效性。在相同的训练条件下，Uni-X相比强大的基线模型在训练效率上表现更优。当扩展到3B参数并使用更大的训练数据时，Uni-X在图像生成任务中达到GenEval 82分，同时在文本和视觉理解任务中也表现出色。这些结果使Uni-X成为未来统一多模态建模的参数高效且可扩展的基础。
英文摘要：
Unified Multimodal Models (UMMs) built on shared autoregressive (AR) transformers are attractive for their architectural simplicity. However, we identify a critical limitation: when trained on multimodal inputs, modality-shared transformers suffer from severe gradient conflicts between vision and text, particularly in shallow and deep layers. We trace this issue to the fundamentally different low-level statistical properties of images and text, while noting that conflicts diminish in middle layers where representations become more abstract and semantically aligned. To overcome this challenge, we propose Uni-X, a two-end-separated, middle-shared architecture. Uni-X dedicates its initial and final layers to modality-specific processing, while maintaining shared parameters in the middle layers for high-level semantic fusion. This X-shaped design not only eliminates gradient conflicts at both ends but also further alleviates residual conflicts in the shared layers. Extensive experiments validate the effectiveness of Uni-X. Under identical training conditions, Uni-X achieves superior training efficiency compared to strong baselines. When scaled to 3B parameters with larger training data, Uni-X matches or surpasses 7B AR-based UMMs, achieving a GenEval score of 82 for image generation alongside strong performance in text and vision understanding tasks. These results establish Uni-X as a parameter-efficient and scalable foundation for future unified multimodal modeling. Our code is available at https://github.com/CURRENTF/Uni-X
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
