arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2505.19084cs.CVcs.AIcs.LG
收藏

Jodi:通过联合建模实现视觉生成与理解的统一

Jodi: Unification of Visual Generation and Understanding via Joint Modeling

Sana

Yifeng Xu, Zhenliang He, Meina Kan, Shiguang Shan, Xilin Chen

AI总结:

提出Jodi扩散框架,通过联合建模图像与标签域,统一视觉生成与理解,支持联合生成、可控生成和图像感知,并构建Joint-1.6M数据集,实验验证其性能与扩展性。

摘要:

视觉生成和理解是人类智能中两个深度互联的方面,然而在机器学习中,它们传统上被视为独立的任务。在本文中,我们提出了Jodi,一个扩散框架,通过联合建模图像域和多个标签域来统一视觉生成与理解。具体来说,Jodi基于线性扩散变换器(linear diffusion transformer)以及角色切换机制构建,使其能够执行三类特定任务:(1)联合生成,即模型同时生成图像和多个标签;(2)可控生成,即基于任意标签组合生成图像;(3)图像感知,即从给定图像中一次性预测多个标签。此外,我们提出了Joint-1.6M数据集,其中包含从公共来源收集的200,000张高质量图像、7个视觉领域的自动标签以及LLM生成的描述。大量实验表明,Jodi在生成和理解任务中均表现出色,并展现出对更广泛视觉领域的强大扩展性。代码可在https://github.com/VIPL-GENUN/Jodi获取。

英文摘要:

Visual generation and understanding are two deeply interconnected aspects of human intelligence, yet they have been traditionally treated as separate tasks in machine learning. In this paper, we propose Jodi, a diffusion framework that unifies visual generation and understanding by jointly modeling the image domain and multiple label domains. Specifically, Jodi is built upon a linear diffusion transformer along with a role switch mechanism, which enables it to perform three particular types of tasks: (1) joint generation, where the model simultaneously generates images and multiple labels; (2) controllable generation, where images are generated conditioned on any combination of labels; and (3) image perception, where multiple labels can be predicted at once from a given image. Furthermore, we present the Joint-1.6M dataset, which contains 200,000 high-quality images collected from public sources, automatic labels for 7 visual domains, and LLM-generated captions. Extensive experiments demonstrate that Jodi excels in both generation and understanding tasks and exhibits strong extensibility to a wider range of visual domains. Code is available at https://github.com/VIPL-GENUN/Jodi.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑