UniFork：探索统一多模态理解与生成的模态对齐
UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation
AI总结：
UniFork提出Y形架构，共享浅层、分支深层，解决统一多模态理解与生成中模态对齐冲突，性能优于全共享Transformer。
摘要：
统一图像理解与生成已成为多模态人工智能中一种有前景的范式。尽管近期取得了进展，但此类统一模型的最优架构设计仍是一个开放的挑战。在本工作中，我们首先分析了用于理解和生成的任务特定专家模型以及当前统一模型的模态对齐行为。我们的分析揭示了一个关键观察：理解任务受益于跨网络深度的模态对齐逐渐增强，这有助于构建语义信息以实现更好的理解；相反，生成任务遵循不同的趋势：模态对齐在早期层增加，但在深层减少以恢复空间细节。这些不同的对齐模式在完全共享的Transformer主干中造成了根本性冲突，其中统一的表征流通常导致两项任务的性能折衷。受此发现启发，我们引入了UniFork，一种新颖的Y形架构，共享浅层以进行跨任务表征学习，同时在深层采用任务特定分支以避免任务干扰。该设计有效平衡了共享学习与任务特化。通过广泛的消融实验，我们证明UniFork持续优于传统的完全共享Transformer架构，并达到与任务特定模型相当或更好的性能。
英文摘要：
Unified image understanding and generation has emerged as a promising paradigm in multimodal artificial intelligence. Despite recent progress, the optimal architectural design for such unified models remains an open challenge. In this work, we start by analyzing the modality alignment behaviors of task-specific expert models for understanding and generation, as well as current unified models. Our analysis reveals a crucial observation: understanding tasks benefit from a progressively increasing modality alignment across network depth, which helps build up semantic information for better comprehension; In contrast, generation tasks follow a different trend: modality alignment increases in the early layers but decreases in the deep layers to recover spatial details. These divergent alignment patterns create a fundamental conflict in fully shared Transformer backbones, where a uniform representational flow often leads to performance compromises across two tasks. Motivated by this finding, we introduce UniFork, a novel Y-shaped architecture that shares the shallow layers for cross-task representation learning, while employing task-specific branches in deeper layers to avoid task interference. This design effectively balances shared learning and task specialization. Through extensive ablation experiments, we demonstrate that Unifork consistently outperforms conventional fully shared Transformer architectures, and achieves performance on par with or better than task-specific models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
