视觉大模型 / VLM

MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings

Haonan Chen, Hong Liu, Yuping Luo, Liang Wang, Nan Yang, Furu Wei, Zhicheng Dou

机构 * Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院） ； Stanford University（斯坦福大学） ； Microsoft Corporation（微软公司） 纠错

专题命中 VLM训练与架构 ：vision language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments Homepage: https://haon-chen.github.io/MoCa/

Multimodal Prompt Alignment for Facial Expression Recognition

Fuyan Ma, Yiran He, Bin Sun, Shutao Li

机构 * Chinese Academy of Military Science（中国军事科学院） ； Changchun University of Science and Technology（长春理工大学） ； Hunan University（湖南大学） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments To appear in ICCV2025

Adding simple structure at inference improves Vision-Language Compositionality

Imanol Miranda, Ander Salaberria, Eneko Agirre, Gorka Azkune

机构 * HiTZ Center – Ixa, University of the Basque Country (UPV/EHU)（希茨中心–Ixa，巴斯克大学（UPV/EHU）） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.LG

LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning

Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, Chongxuan Li

机构 * Gaoling School of AI, Renmin University of China（中国人民大学人工智能学院） ； Beijing Key Laboratory of Research on Large Models and Intelligent Governance（北京大型模型与智能治理研究重点实验室） ； Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE（下一代智能搜索与推荐工程技术研究中心，教育部） ； Ant Group（蚂蚁集团） 纠错

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.LG

Comments Project page and codes: \url{https://ml-gsai.github.io/LLaDA-V-demo/}

Improve Multi-Modal Embedding Learning via Explicit Hard Negative Gradient Amplifying

Youze Xue, Dian Li, Gang Liu

机构 * Tencent（腾讯） 纠错

专题命中 VLM训练与架构 ：LLaVA(abstract)；MLLM(abstract)；分类 cs.CV、cs.LG

Dual-Process Image Generation

Grace Luo, Jonathan Granskog, Aleksander Holynski, Trevor Darrell

机构 * UC Berkeley（伯克利大学） ； Runway 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Mixpert: Mitigating Multimodal Learning Conflicts with Efficient Mixture-of-Vision-Experts

Xin He, Xumeng Han, Longhui Wei, Lingxi Xie, Qi Tian

机构 * Huawei Inc.（华为公司） ； University of Chinese Academy of Sciences（中国科学院大学） 纠错

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

VITA: Towards Open-Source Interactive Omni Multimodal LLM

Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Yuhang Dai, Meng Zhao, Yi-Fan Zhang, Shaoqi Dong, Yangze Li, Xiong Wang, Haoyu Cao, Di Yin, Long Ma, Xiawu Zheng, Rongrong Ji, Yunsheng Wu, Ran He, Caifeng Shan, Xing Sun

机构 * NJU（南京大学） ； Tencent Youtu Lab（腾讯计算机系统有限公司视觉实验室） ； XMU（厦门大学） ； CASIA（中国科学院自动化研究所） 纠错

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments Project Page: https://vita-home.github.io

QuARI: Query Adaptive Retrieval Improvement

Eric Xing, Abby Stylianou, Robert Pless, Nathan Jacobs

机构 * Washington University in St. Louis（华盛顿大学圣路易斯分校） ； Saint Louis University（圣路易斯大学） ； The George Washington University（乔治华盛顿大学） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Comments 13 pages, 4 figures, 4 tables

CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling

Jihai Zhang, Xiaoye Qu, Tong Zhu, Yu Cheng

机构 * First Author Affiliation（第一作者机构） ； Second Author Affiliation（第二作者机构） 纠错

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Multimodal LLM-Guided Semantic Correction in Text-to-Image Diffusion

Zheqi Lv, Junhao Chen, Qi Tian, Keting Yin, Shengyu Zhang, Fei Wu

机构 * Zhejiang University（浙江大学） 纠错

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attention

Yucheng Li, Huiqiang Jiang, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Amir H. Abdi, Dongsheng Li, Jianfeng Gao, Yuqing Yang, Lili Qiu

专题命中 VLM训练与架构 ：vision language model(abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Comments Accepted at ICML 2025

Multi-modal Synthetic Data Training and Model Collapse: Insights from VLMs and Diffusion Models

Zizhao Hu, Mohammad Rostami, Jesse Thomason

机构 * University of Southern California（南加州大学） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI、cs.LG

2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining

Wenqi Zhang, Hang Zhang, Xin Li, Jiashuo Sun, Yongliang Shen, Weiming Lu, Deli Zhao, Yueting Zhuang, Lidong Bing

机构 * College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院） ； DAMO Academy, Alibaba Group（阿里巴巴集团达摩院） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Comments Under review

DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMs

Zhenhailong Wang, Senthil Purushwalkam, Caiming Xiong, Silvio Savarese, Heng Ji, Ran Xu

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） ； Salesforce Research（Salesforce 研究） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Facilitating Video Story Interaction with Multi-Agent Collaborative System

Yiwen Zhang, Jianing Hao, Zhan Wang, Hongling Sheng, Wei Zeng

机构 * The Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州）） 纠错

专题命中 VLM训练与架构 ：vision language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments Prepared and submitted in 2024

Topology-Aware CLIP Few-Shot Learning

Dazhi Huang

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Token-Level Prompt Mixture with Parameter-Free Routing for Federated Domain Generalization

Shuai Gong, Chaoran Cui, Xiaolin Dong, Xiushan Nie, Lei Zhu, Xiaojun Chang

机构 * School of Computing and Artificial Intelligence, Shandong University of Finance and Economics（计算机与人工智能学院，山东财经大学） ； School of Computer Science and Technology, Shandong Jianzhu University（计算机科学与技术学院，山东建筑大学） ； College of Electronics and Information Engineering, Tongji University（电子信息工程学院，同济大学） ； School of Information Science and Technology, University of Science and Technology of China（信息科学与技术学院，中国科学技术大学） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI、cs.LG

Comments The manuscript has been submitted to IEEE Transactions on Knowledge and Data Engineering

Causal Graphical Models for Vision-Language Compositional Understanding

Fiorenzo Parascandolo, Nicholas Moratelli, Enver Sangineto, Lorenzo Baraldi, Rita Cucchiara

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments Accepted at ICLR 2025

Self-Evolving Visual Concept Library using Vision-Language Critics

Atharva Sehgal, Patrick Yuan, Ziniu Hu, Yisong Yue, Jennifer J. Sun, Swarat Chaudhuri

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Comments CVPR camera ready

Astrea: A MOE-based Visual Understanding Model with Progressive Alignment

Xiaoda Yang, JunYu Lu, Hongshun Qiu, Sijing Li, Hao Li, Shengpeng Ji, Xudong Tang, Jiayang Xu, Jiaqi Duan, Ziyue Jiang, Cong Lin, Sihang Cai, Zejian Xie, Zhuoyang Song, Songxin Zhang

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Tokenization of Gaze Data

Tim Rolff, Jurik Karimian, Niklas Hypki, Susanne Schmidt, Markus Lappe, Frank Steinicke

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.LG

Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection

Adyasha Maharana, Jaehong Yoon, Tianlong Chen, Mohit Bansal

专题命中 VLM训练与架构 ：grounding(abstract)；multimodal large language model(abstract)；分类 cs.AI、cs.LG

Comments First two authors contributed equally. Code: https://github.com/adymaharana/adapt-inf

Debugging and Runtime Analysis of Neural Networks with VLMs (A Case Study)

Boyue Caroline Hu, Divya Gopinath, Corina S. Pasareanu, Nina Narodytska, Ravi Mangal, Susmit Jha

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI、cs.LG

Comments CAIN 2025 (4th International Conference on AI Engineering -- Software Engineering for AI)

GiVE: Guiding Visual Encoder to Perceive Overlooked Information

Junjie Li, Jianghong Ma, Xiaofeng Zhang, Yuhang Li, Jianyang Shi

专题命中 VLM训练与架构 ：visual question answering(abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments This paper was accepted by ICME 2025

MMLNB: Multi-Modal Learning for Neuroblastoma Subtyping Classification Assisted with Textual Description Generation

Huangwei Chen, Yifei Chen, Zhenyu Yan, Mingyang Ding, Chenlei Li, Zhu Zhu, Feiwei Qin

专题命中 VLM训练与架构 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments 25 pages, 7 figures

BREEN: Bridge Data-Efficient Encoder-Free Multimodal Learning with Learnable Queries

Tianle Li, Yongming Rao, Winston Hu, Yu Cheng

专题命中 VLM训练与架构 ：InternVL(abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search

Yiming Jia, Jiachen Li, Xiang Yue, Bo Li, Ping Nie, Kai Zou, Wenhu Chen

专题命中 VLM训练与架构 ：vision-language model(abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

Comments Technical Report

A Two-Step Concept-Based Approach for Enhanced Interpretability and Trust in Skin Lesion Diagnosis

Cristiano Patrício, Luís F. Teixeira, João C. Neves

专题命中 VLM训练与架构 ：vision language model(abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Comments Published in the Computational and Structural Biotechnology Journal

Is Your Video Language Model a Reliable Judge?

Ming Liu, Wensheng Zhang

专题命中 VLM训练与架构 ：VLM(abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI