arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.04423cs.CVcs.AI
收藏

统一多模态模型中理解与生成之间的可迁移性

Transferability Between Understanding and Generation in Unified Multimodal Models

KAIST AI(韩国科学技术院人工智能研究所);Blynx(布林克斯公司);Trillionlabs(万亿实验室)

Jiwon Kang, Heeji Yoon, Jaewoo Jung, Jaewon Min, Minkyeong Jeon, Biyeon Hwang, Sangwon Jung, Seungryong Kim

AI总结:

研究统一多模态模型中理解与生成任务间的可迁移性,通过实验发现其依赖架构,据此提出实用训练策略,训练对应理解任务迁移至生成可提升性能并减少分布偏移。

摘要:

统一多模态模型(UMMs)在单一架构中整合图像理解与生成,但两者如何交互仍研究不足。我们研究UMMs中的可迁移性:在一项任务上训练一种能力是否能在无明确监督的情况下提升另一任务的相同能力。通过控制实验,我们实证发现可迁移性取决于架构——具有完全共享变压器主干和统一视觉编码器的模型表现出一致的跨任务迁移,而松散耦合设计则很少或没有。利用这种可迁移性,我们提出了一种实用的训练策略。提高目标生成能力(例如计数)最直接的方法是直接微调生成,但这可能由于分布偏移而降低视觉质量。相反,我们训练相应的理解任务并让其迁移到生成中,这在最小化分布偏移的同时提高了特定能力的生成性能。我们在三种能力——计数、空间关系和文本识别/生成——上验证了这一点,表明跨任务可迁移性可以在UMMs中得到系统利用。

英文摘要:

Unified Multimodal Models (UMMs) integrate image understanding and generation within a single architecture, yet how the two tasks interact remains understudied. We investigate $\boldsymbol{\mathsf{transferability}}$ in UMMs: whether training a capability on one task improves the same capability on the other without explicit supervision. Through controlled experiments, we empirically find that transferability depends on architecture-models with fully shared transformer backbone and a unified visual encoder exhibit consistent cross-task transfer, while loosely coupled designs show little or none. Leveraging this transferability, we propose a practical training strategy. The most straightforward way to improve a target generative capability (e.g., counting) is to fine-tune generation directly, but this can degrade visual quality due to distribution shift. Instead, we train the corresponding understanding task and let it transfer into generation, which improves capability-specific generative performance while minimizing distribution shift. We validate this across three capabilities-counting, spatial relation, and text recognition/generation-showing that cross-task transferability can be systematically exploited in UMMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑