Unified Multimodal Models (UMMs) integrate image understanding and generation within a single architecture, yet how the two tasks interact remains understudied. We investigate $\boldsymbol{\mathsf{transferability}}$ in UMMs: whether training a capability on one task improves the same capability on the other without explicit supervision. Through controlled experiments, we empirically find that transferability depends on architecture-models with fully shared transformer backbone and a unified visual encoder exhibit consistent cross-task transfer, while loosely coupled designs show little or none. Leveraging this transferability, we propose a practical training strategy. The most straightforward way to improve a target generative capability (e.g., counting) is to fine-tune generation directly, but this can degrade visual quality due to distribution shift. Instead, we train the corresponding understanding task and let it transfer into generation, which improves capability-specific generative performance while minimizing distribution shift. We validate this across three capabilities-counting, spatial relation, and text recognition/generation-showing that cross-task transferability can be systematically exploited in UMMs.
Transfer Attack for Bad and Good: Explain and Boost Adversarial Transferability across Multimodal Large Language Models
The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));University of Oxford(牛津大学);Drexel University(德雷塞尔大学);Beijing University of Technology(北京理工大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Xi’an Jiaotong University(西安交通大学)
Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction
School of Computer Science, Fudan University(复旦大学计算机科学学院);Academy for Engineering & Technology, Fudan University(复旦大学工程技术学院);Shanghai Key Lab of Intelligent Information Processing(上海智能信息处理重点实验室);Shanghai Engineering Research Center of AI & Robotics(上海人工智能与机器人工程研究中心);Engineering Research Center of AI & Robotics, Ministry of Education(教育部人工智能与机器人工程研究中心)
Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models
Southern University of Science and Technology(南方科技大学);The University of Hong Kong(香港大学);Monash University(莫纳什大学);Temple University(天普大学);Peng Cheng Laboratory(鹏城实验室)
Geometric Understanding of Discriminability and Transferability for Visual Domain Adaptation
School of Mathematics, Sun Yat-Sen University(中山大学数学学院);School of Statistics and Mathematics, Guangdong University of Finance and Economics(广东财经大学统计与数学学院);School of Computer Science, Nanjing University of Posts and Telecommunications(南京邮电大学计算机学院)
Towards Understanding and Boosting Adversarial Transferability from a Distribution Perspective
Zhejiang University(浙江大学);Alibaba Group(阿里巴巴集团);University of Science and Technology of China(中国科学技术大学);Hangzhou Dianzi University(杭州电子科技大学);University of Chinese Academy of Sciences(中国科学院大学);Chinese Academy of Sciences(中国科学院)
HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generation
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院,清华大学);ARC Lab, Tencent PCG(腾讯PCG ARC实验室);The University of Hong Kong(香港大学);Xi’an JiaoTong University(西安交通大学)
ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability
Nankai University(南开大学);Shanghai Innovation Institute(上海创新研究院);University of Science and Technology of China(中国科学技术大学);Wuhan University(武汉大学);Shanghai AI Laboratory(上海人工智能实验室)
Why Does Little Robustness Help? A Further Step Towards Understanding Adversarial Transferability
School of Cyber Science and Engineering, Huazhong University of Science and Technology(电子科学与工程学院,华中科技大学);School of Software Engineering, Huazhong University of Science and Technology(软件工程学院,华中科技大学);School of Information and Communication Technology, Griffith University(信息与通信技术学院,格里菲斯大学);School of Computer Science and Technology, Huazhong University of Science and Technology(计算机科学与技术学院,华中科技大学)
Understanding vs. Generation: Navigating Optimization Dilemma in Multimodal Models
State Key Laboratory of General Artificial Intelligence, Peking University(北京大学通用人工智能国家重点实验室);Tencent(腾讯);Center for Data Science, Peking University(北京大学数据科学中心);Center for Machine Learning Research, Peking University(北京大学机器学习研究中心)
EHWGesture -- A dataset for multimodal understanding of clinical gestures
Department of Control and Computer Engineering, Politecnico di Torino(控制与计算机工程系,都灵理工大学);Fondazione Bruno Kessler(布鲁诺·凯斯勒基金会);CNR-IEIIT(意大利国家研究委员会-IEIIT)
Improving Adversarial Transferability on Vision-Language Pre-training Models via Surrogate-Specific Bias Correction
School of Cyber Science and Engineering, Southeast University(东南大学网络空间安全学院);Purple Mountain Laboratories(紫金山实验室);School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院)
Revisiting the Transferability of Supervised Pretraining: an MLP Perspective
Zhejiang University(浙江大学);The University of Sydney(悉尼大学);SenseTime Research(商汤科技研究院);Qing Yuan Research Institute, Shanghai Jiao Tong University(上海交通大学清源研究院)
SurgLLM: A Versatile Large Multimodal Model with Spatial Focus and Temporal Awareness for Surgical Video Understanding
Hong Kong Institute of Science & Innovation(香港科学与工业创新研究院);CAMP, Technische Universität München(CAMP,慕尼黑技术大学);Department of Surgery, Faculty of Medicine, The Chinese University of Hong Kong(香港中文大学医学院外科部)
New Ideas and Trends in Deep Multimodal Content Understanding: A Review
LIACS, Leiden University(莱顿大学高级计算机科学研究所);College of Systems Engineering, NUDT(国防科技大学系统工程学院);Center for Machine Vision and Signal Analysis, University of Oulu(奥卢大学机器视觉与信号分析中心)
School of Computer Science and Engineering, Southeast University, Nanjing, China(计算机科学与工程学院,东南大学,南京,中国);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其跨学科应用关键实验室(东南大学),教育部,中国)
针对多模态视频模型中强模态主导导致的过拟合与模态不平衡问题,提出 MM 及改进的 B-MM 方法,通过动态调整模态混合比例提升泛化能力与鲁棒性。
LVRPO: Language-Visual Alignment with GRPO for Multimodal Understanding and Generation
Department of Machine Learning, CMU, USA(卡内基梅隆大学机器学习系,美国);Department of Machine Learning, MBZUAI, UAE(穆罕默德·本·扎耶德人工智能大学机器学习系,阿联酋);Department of Artificial Intelligence, Hanyang University ERICA, South Korea(汉阳大学ERICA校区人工智能系,韩国)
Reversing the Flow: Generation-to-Understanding Synergy in Large Multimodal Models
Beijing University of Posts and Telecommunications(北京邮电大学);Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance(北京多模态数据智能感知与治理重点实验室)
GPT-4o: Visual perception performance of multimodal large language models in piglet activity understanding
China Agricultural University(中国农业大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);AniEye Technology(安尼眼科技);State Key Laboratory of Animal Nutrition and Feeding(动物营养与饲料国家重点实验室)