视觉大模型 / VLM

A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges

Zongxia Li, Xiyang Wu, Hongyang Du, Fuxiao Liu, Huy Nghiem, Guangyao Shi

专题命中 VLM训练与架构 ：vision language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 22 pages, 3 figures

Journal ref Navigating the Future: Ensuring Trustworthiness in Multi-Modal Open-World Intelligence @ CVPR, 2025

Advancing Chronic Tuberculosis Diagnostics Using Vision-Language Models: A Multi modal Framework for Precision Analysis

Praveen Shastry, Sowmya Chowdary Muthulur, Naveen Kumarasami, Anandakumar D, Mounigasri M, Keerthana R, Kishore Prasath Venkatesh, Bargava Subramanian, Kalyan Sivasailam, Revathi Ezhumalai, Abitha Marimuthu

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 10 pages , 3 figures

Advancing vision-language models in front-end development via data synthesis

Tong Ge, Yashu Liu, Jieping Ye, Tianyi Li, Chao Wang

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

How Culturally Aware are Vision-Language Models?

Olena Burda-Lassen, Aman Chadha, Shashank Goswami, Vinija Jain

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI、cs.LG

Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models

Zhiqi Li, Guo Chen, Shilong Liu, Shihao Wang, Vibashan VS, Yishen Ji, Shiyi Lan, Hao Zhang, Yilin Zhao, Subhashree Radhakrishnan, Nadine Chang, Karan Sapra, Amala Sanjay Deshmukh, Tuomas Rintamaki, Matthieu Le, Ilia Karmanov, Lukas Voegtle, Philipp Fischer, De-An Huang, Timo Roman, Tong Lu, Jose M. Alvarez, Bryan Catanzaro, Jan Kautz, Andrew Tao, Guilin Liu, Zhiding Yu

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

Visual RAG: Expanding MLLM visual knowledge without fine-tuning

Mirco Bonomo, Simone Bianco

专题命中 VLM训练与架构 ：MLLM(title,abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI、cs.LG

PaliGemma: A versatile 3B VLM for transfer

Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, Xiaohua Zhai

专题命中 VLM训练与架构 ：VLM(title,abstract)；vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments v2 adds Appendix H and I and a few citations

Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning

Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, Sergey Levine

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

SQ-LLaVA: Self-Questioning for Large Vision-Language Assistant

Guohao Sun, Can Qin, Jiamian Wang, Zeyuan Chen, Ran Xu, Zhiqiang Tao

专题命中 VLM训练与架构 ：LLaVA(title,abstract)；vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments ECCV 2024

Visual Language Model based Cross-modal Semantic Communication Systems

Feibo Jiang, Chuanguo Tang, Li Dong, Kezhi Wang, Kun Yang, Cunhua Pan

专题命中 VLM训练与架构 ：visual language model(title)；vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 12 pages, 10 figures

Ovis: Structural Embedding Alignment for Multimodal Large Language Model

Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Han-Jia Ye

专题命中 VLM训练与架构 ：multimodal large language model(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI、cs.LG

Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models

Christian Schlarmann, Naman Deep Singh, Francesco Croce, Matthias Hein

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments ICML 2024 Oral

Promoting AI Equity in Science: Generalized Domain Prompt Learning for Accessible VLM Research

Qinglong Cao, Yuntian Chen, Lu Lu, Hao Sun, Zhenzhong Zeng, Xiaokang Yang, Dongxiao Zhang

专题命中 VLM训练与架构 ：VLM(title,abstract)；vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

BRAVE: Broadening the visual encoding of vision-language models

Oğuzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, Federico Tombari

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Project page at https://brave-vlms.epfl.ch/

Representing Online Handwriting for Recognition in Large Vision-Language Models

Anastasiia Fadeeva, Philippe Schlattner, Andrii Maksai, Mark Collier, Efi Kokiopoulou, Jesse Berent, Claudiu Musat

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection

Ruibo Chen, Yihan Wu, Lichang Chen, Guodong Liu, Qi He, Tianyi Xiong, Chenxi Liu, Junfeng Guo, Heng Huang

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；LLaVA(abstract)

Comments 9 pages, 3 figures, 4 tables

Linear Spaces of Meanings: Compositional Structures in Vision-Language Models

Matthew Trager, Pramuditha Perera, Luca Zancato, Alessandro Achille, Parminder Bhatia, Stefano Soatto

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 18 pages, 9 figures, 7 tables

Journal ref Proceedings of the IEEE/CVF International Conference on Computer Vision 2023 (pp. 15395-15404)

From Understanding to Engagement: Personalized pharmacy Video Clips via Vision Language Models (VLMs)

从理解到参与：通过视觉语言模型（VLMs）生成个性化药学视频片段

Suyash Mishra, Qiang Li, Srikanth Patil, Anubhav Girdhar

机构 * Roche（罗氏） ； Accenture（埃森哲） ； Involead 纠错

专题命中 VLM训练与架构 ：vision language model(title,abstract)；VLM(abstract,comments)；分类 cs.CV、cs.LG

AI总结 本文提出基于视觉语言模型和音频语言模型的个性化药学视频片段生成方法，通过剪切合并算法和个性化机制提升生成效率与质量，实现制药行业内容处理的高效自动化。

Comments Contributed original research to top tier conference in VLM; currently undergoing peer review

Face-LLaVA: Facial Expression and Attribute Understanding through Instruction Tuning

Ashutosh Chaubey, Xulang Guan, Mohammad Soleymani

专题命中 VLM训练与架构 ：LLaVA(title,abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments Project Page: https://face-llava.github.io

Matryoshka Query Transformer for Large Vision-Language Models

Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng, Kai-Wei Chang

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；LLaVA(abstract,comments)；分类 cs.CV、cs.LG

Comments Preprint. Our code and model are publicly available at https://github.com/gordonhu608/MQT-LLaVA

Language Models as Black-Box Optimizers for Vision-Language Models

Shihong Liu, Zhiqiu Lin, Samuel Yu, Ryan Lee, Tiffany Ling, Deepak Pathak, Deva Ramanan

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract,comments)；分类 cs.CV、cs.LG

Comments Published at CVPR 2024. Project site: https://llm-can-optimize-vlm.github.io/

CL4D: Contrastive Language-4D Pretraining for Vision-Language Reasoning in Dynamic Scenes

CL4D：用于动态场景视觉-语言推理的对比语言-4D预训练

Kumal Hewagamage, Isuranga Senavirathne, Sasika Amarasinghe, Hasitha Gallella, Dulanga Weerakoon, Vigneshwaran Subbaraju, Ranga Rodrigo

机构 * University of Moratuwa（莫拉图瓦大学） ； Singapore-MIT Alliance for Research & Technology (SMART) Centre（新加坡-麻省理工研究与技术联盟（SMART）中心） ； Agency for Science, Technology and Research (A*STAR)（新加坡科学、技术与研究局（A*STAR）） 纠错

专题命中 VLM训练与架构 ：VLM(summary_cn,abstract)；vision-language model(abstract)；分类 cs.CV

AI总结 本研究提出CL4D（首个4D视觉编码器）及基于其的4DVLM，在自建DynAction4D数据集上训练，CL4D性能较现有方法提升约16.75%，4DVLM优于Gemini、GPT-5等前沿视频VLM。

Comments Accepted at the 19th European Conference on Computer Vision (ECCV 2026)

VLCP: Vision Language Control Policy Closed-Loop Code Replanning for Robot Manipulation

VLCP：用于机器人操纵的视觉语言控制策略闭环代码重规划

Dhia Naouali, Minghan Wu, Claudia Wong, Abhinav Puthran, Omar G. Younis

机构 * University of Monastir（莫纳斯提尔大学） ； St. Mildred’s-Lightbourn School（圣米尔德雷德-莱特本学校） ； Cornell University（康奈尔大学） ； Carnegie Mellon University（卡内基梅隆大学） ； Silverstream AI（银流人工智能公司） ； Mila -- Quebec AI Institute（米拉-魁北克人工智能研究所） 纠错

专题命中 VLM训练与架构 ：VLM(summary_cn,abstract)；vision-language model(abstract)；分类 cs.LG

AI总结 VLCP是一种无需训练的机器人操纵策略，通过在单回合内每K步由VLM重写控制代码闭合循环，在57项任务的评估中综合成功率达35.1%，较开环策略提升十倍。

LIRA: Local Cross-Layer Information Routing for Vision-Language-Action Decoding

LIRA：面向视觉-语言-动作解码的局部跨层信息路由

Zhewei Zhang, Puyue Wang, Guanren Qiao, Yijie Weng, Jiawei Hu, Guo Li, Lujia Wang, Junyan Wang, Tao Gu, Hongliang Lu, Guiliang Liu, Hong Jia, Xinhu Zheng

专题命中 VLM训练与架构 ：VLM(summary_cn,abstract)；vision-language model(abstract)；分类 cs.CV

AI总结 LIRA是面向VLA模型的局部跨层信息路由机制，通过深度感知路由VLM特征，在多机器人操作基准及真实场景中提升了动作预测性能与分布偏移下的鲁棒性。

Comments 9 pages, 4 figures. Code and model checkpoints will be released upon acceptance of the paper

VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression

VisCo：利用大语言模型作为视觉令牌压缩的内在编码器

Yupeng Zheng, Kai Zou, Bin Liu, Nenghai Yu

专题命中 VLM训练与架构 ：VLM(summary_cn,abstract)；vision-language model(abstract)；分类 cs.CV

AI总结 研究针对视觉语言模型处理视觉令牌时的高成本问题，提出训练高效的自压缩框架VisCo，将预训练VLM用作内在压缩器，通过参数共享自动编码器压缩视觉信息，实验证明其在压缩率上超先前方法，还能捕获互补表示改进基础模型。

Comments Accepted by ACM MM 2026

RUTA: Principled Visual Token Allocation via Rate-Utility Optimization

RUTA：基于率-效用优化的原则性视觉令牌分配方法

Jian Zou, Xiaoyu Xu, Zhihua Wang, Yilin Wang, Balu Adsumilli, Kede Ma

机构 * City University of Hong Kong（香港城市大学） ； Google Inc.（谷歌公司） 纠错

专题命中 VLM训练与架构 ：LLaVA(summary_cn,abstract)；vision-language model(abstract)；分类 cs.CV

AI总结 针对视觉语言模型因长视觉令牌序列导致的LLM侧计算和内存成本过高问题，提出RUTA方法，仅用2.0%、4.2%的视觉令牌，分别在LLaVA-NeXT-7B、Qwen3-VL-8B上保留88.2%、94.4%的任务性能。

Taming the Implicit: Dual-Channel Risk-Aware Reinforcement Fine-Tuning for Continual Multimodal Post-Training

驯服隐式：面向持续多模态后训练的双通道风险感知强化微调

Yibei Liu, Jiajun Chen, Qianle Zhang, Tangyue Jin, Mengying Zhu, Meng Xi, Yangyang Wu

专题命中 VLM训练与架构 ：MLLM(summary_cn,abstract)；multimodal large language model(abstract)；分类 cs.AI

AI总结 针对持续多模态后训练中RFT算法在任务分布偏移下遗忘加剧的问题，提出双通道风险感知强化微调框架RAPO，通过策略与数据通道的风险管控降低遗忘，在MLLM-CL基准上使遗忘减少79.8%且保留新任务竞争力。

MIEScore: Human-Aligned Evaluation for Multi-Source Image Editing

MIEScore：面向多源图像编辑的人类对齐评估方法

Zitong Xu, Huiyu Duan, Xinyun Zhang, Weifei Xiong, Tianyi Zheng, Xiongkuo Min, Qiang Hu, Zhengxue Cheng, Bo Li, Guangtao Zhai

专题命中 VLM训练与架构 ：MLLM(summary_cn,abstract)；multimodal large language model(abstract)；分类 cs.CV

AI总结 针对多源图像编辑（MIE）缺乏人类对齐评估基准的问题，研究构建了首个MIE基准MIE-Bench，并提出基于MLLM的评估模型MIEScore，其对齐人类偏好性能最优且泛化性良好。

MoRoute: Dynamic Routing for In-Context Multimodal Video Generation

MoRoute：面向上下文多模态视频生成的动态路由

Chong Gao, Jie Ma, Zhan Peng, Chongxiao Wang, Haoxue Wu, Jun Liang, Guanbin Li, Jing Li

机构 * Sun Yat-sen University（中山大学） ； HUJING Digital Media & Entertainment Group（沪景数字媒体娱乐集团） ； Huazhong University of Science and Technology（华中科技大学） 纠错

专题命中 VLM训练与架构 ：VLM(summary_cn,abstract)；vision-language model(abstract)；分类 cs.CV

AI总结 MoRoute是采用动态层路由连接VLM与视频DiT的多模态视频生成框架，在三个基准数据集上均优于现有最优方法，提升了视频生成与编辑的性能。

Comments Project page: https://orange-3dv-team.github.io/MoRoute/

WorldDiT: A Unified Diffusion Architecture for World and Action Modeling

WorldDiT：用于世界和动作建模的统一扩散架构

Sen Wang, R. Gnana Praveen, Bidhan Roy, Marcos Villagra

专题命中 VLM训练与架构 ：VLM(summary_cn,abstract)；vision-language model(abstract)；分类 cs.LG

AI总结 研究提出WorldDiT统一扩散架构，结合动作生成与视觉世界建模，不依赖大型预训练VLM动作主干，在四个LIBERO模拟套件中表现出色，处于帕累托前沿，为未来扩展研究提供了低于十亿参数的基线。

Comments 9 pages, 4 figures