Visual generation and understanding are two deeply interconnected aspects of human intelligence, yet they have been traditionally treated as separate tasks in machine learning. In this paper, we propose Jodi, a diffusion framework that unifies visual generation and understanding by jointly modeling the image domain and multiple label domains. Specifically, Jodi is built upon a linear diffusion transformer along with a role switch mechanism, which enables it to perform three particular types of tasks: (1) joint generation, where the model simultaneously generates images and multiple labels; (2) controllable generation, where images are generated conditioned on any combination of labels; and (3) image perception, where multiple labels can be predicted at once from a given image. Furthermore, we present the Joint-1.6M dataset, which contains 200,000 high-quality images collected from public sources, automatic labels for 7 visual domains, and LLM-generated captions. Extensive experiments demonstrate that Jodi excels in both generation and understanding tasks and exhibits strong extensibility to a wider range of visual domains. Code is available at https://github.com/VIPL-GENUN/Jodi.
Crab$^{+}$: A Scalable and Unified Audio-Visual Scene Understanding Model with Explicit Cooperation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学耿丽人工智能学院);Institute of Artificial Intelligence of China Telecom (TeleAI)(中国电信人工智能研究院);AI Technology Center, Online Video Business Unit, Tencent PCG(腾讯PCG在线视频业务单元AI技术中心);Hefei University of Technology(合肥工业大学);The University of Hong Kong(香港大学)
Unified Diffusion VLA: Vision-Language-Action Model via Joint Discrete Denoising Diffusion Process
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Westlake University(西交大学);Zhejiang University(浙江大学);Monash University(墨尔本大学)
UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions
State Key Laboratory for Novel Software Technology(新型软件技术国家重点实验室);Tencent Hunyuan(腾讯文英);Shanghai Jiao Tong University(上海交通大学);Renmin University of China(中国人民大学);Tsinghua University(清华大学);Shanghai AI Lab(上海人工智能实验室)
MIGE: Mutually Enhanced Multimodal Instruction-Based Image Generation and Editing
State Key Laboratory of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(人工智能安全国家重点实验室,计算技术研究所,中国科学院);University of Chinese Academy of Sciences(中国科学院大学)
Lumina-Image 2.0: A Unified and Efficient Image Generative Framework
Shanghai AI Laboratory(上海人工智能实验室);The University of Sydney(悉尼大学);The Chinese University of Hong Kong(香港中文大学);Shanghai Innovation Institute(上海创新研究院);Shanghai Jiao Tong University(上海交通大学);Krea AI(Krea AI公司)
UniEmo: Unifying Emotional Understanding and Generation with Learnable Expert Queries
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Great Bay University(大湾区大学);School of Computing and Information Technology(计算机与信息学院);Dongguan Key Laboratory for Intelligence and Information Technology(东莞智能与信息科技重点实验室);Shenzhen Loop Area Institute(深圳河套学院);Macao Polytechnic University(澳门 polytechnic 大学)