视觉大模型 / VLM - arXivDaily 专题

2405.06634 2024-06-11 cs.CV cs.AI cs.CL 73%

Multimodal LLMs Struggle with Basic Visual Network Analysis: a VNA Benchmark

Evan M. Williams, Kathleen M. Carley

专题命中 VLM训练与架构：vision language model(abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

Comments 11 pages, 3 figures

2406.00872 2024-06-04 cs.CV cs.AI 73%

OLIVE: Object Level In-Context Visual Embeddings

Timothy Ossowski, Junjie Hu

专题命中 VLM训练与架构：vision-language model(abstract)；grounding(abstract)；分类 cs.CV、cs.AI

Comments ACL 2024

2312.14867 2024-06-04 cs.CV cs.AI cs.CL cs.MM 73%

VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation

Max Ku, Dongfu Jiang, Cong Wei, Xiang Yue, Wenhu Chen

专题命中 VLM训练与架构：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments Accepted to ACL2024 main

2403.04822 2024-05-28 cs.CV cs.LG 73%

UniTable: Towards a Unified Framework for Table Recognition via Self-Supervised Pretraining

ShengYun Peng, Aishwarya Chakravarthy, Seongmin Lee, Xiaojing Wang, Rajarajeswari Balasubramaniyan, Duen Horng Chau

专题命中 VLM训练与架构：vision-language model(abstract)；LLaVA(abstract)；分类 cs.CV、cs.LG

2405.11273 2024-05-21 cs.AI cs.CL cs.CV cs.MM 73%

Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts

Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang, Wanqi Zhong, Wenhan Luo, Lin Ma, Min Zhang

专题命中 VLM训练与架构：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments 22 pages, 13 figures. Project Website: https://uni-moe.github.io/. Working in progress

2404.09619 2024-04-16 cs.CV cs.AI 73%

UNIAA: A Unified Multi-modal Image Aesthetic Assessment Baseline and Benchmark

Zhaokun Zhou, Qiulin Wang, Bin Lin, Yiwei Su, Rui Chen, Xin Tao, Amin Zheng, Li Yuan, Pengfei Wan, Di Zhang

专题命中 VLM训练与架构：LLaVA(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

2404.04763 2024-04-09 cs.CV cs.AI 73%

GenEARL: A Training-Free Generative Framework for Multimodal Event Argument Role Labeling

Hritik Bansal, Po-Nien Kung, P. Jeffrey Brantingham, Kai-Wei Chang, Nanyun Peng

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments 20 pages, 15 Figures, 13 figures

2404.03590 2024-04-05 cs.CV cs.AI 73%

SemGrasp: Semantic Grasp Generation via Language Aligned Discretization

Kailin Li, Jingbo Wang, Lixin Yang, Cewu Lu, Bo Dai

专题命中 VLM训练与架构：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

2404.01156 2024-04-02 cs.CV cs.AI 73%

SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining

Chull Hwan Song, Taebaek Hwang, Jooyoung Yoon, Shunghyun Choi, Yeong Hyeon Gu

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments CVPR2024 Accepted

2303.05983 2024-02-20 cs.CV cs.AI 73%

Accountable Textual-Visual Chat Learns to Reject Human Instructions in Image Re-creation

Zhiwei Zhang, Yuliang Liu

专题命中 VLM训练与架构：VLM(abstract)；visual language model(abstract)；分类 cs.CV、cs.AI

Comments TMLR, Survey Certification

2401.02137 2024-01-05 cs.CV cs.AI 73%

SyCoCa: Symmetrizing Contrastive Captioners with Attentive Masking for Multimodal Alignment

Ziping Ma, Furong Xu, Jian Liu, Ming Yang, Qingpei Guo

专题命中 VLM训练与架构：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

2307.04317 2023-10-31 cs.CV cs.LG 73%

Text Descriptions are Compressive and Invariant Representations for Visual Learning

Zhili Feng, Anna Bair, J. Zico Kolter

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.LG

2309.13430 2023-09-26 cs.CL cs.AI cs.CV 73%

Resolving References in Visually-Grounded Dialogue via Text Generation

Bram Willemsen, Livia Qian, Gabriel Skantze

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments Published at SIGDIAL 2023

2308.12270 2023-08-24 cs.LG cs.AI 73%

Language Reward Modulation for Pretraining Reinforcement Learning

Ademi Adeniji, Amber Xie, Carmelo Sferrazza, Younggyo Seo, Stephen James, Pieter Abbeel

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.AI、cs.LG

Comments Code available at https://github.com/ademiadeniji/lamp

2306.05425 2023-06-09 cs.CV cs.AI cs.CL cs.HC 73%

MIMIC-IT: Multi-Modal In-Context Instruction Tuning

Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, Ziwei Liu

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments Project page: https://otter-ntu.github.io/ Dataset & code: https://github.com/Luodian/otter Initial release, work in progress

2208.13628 2022-10-06 cs.CV cs.LG 73%

Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment

Mustafa Shukor, Guillaume Couairon, Matthieu Cord

专题命中 VLM训练与架构：visual reasoning(abstract)；grounding(abstract)；分类 cs.CV、cs.LG

Comments BMVC 2022

2004.12070 2020-10-13 cs.CV cs.CL cs.LG 73%

Deep Multimodal Neural Architecture Search

Zhou Yu, Yuhao Cui, Jun Yu, Meng Wang, Dacheng Tao, Qi Tian

专题命中 VLM训练与架构：visual question answering(abstract)；grounding(abstract)；分类 cs.CV、cs.LG

Comments Accept to ACM MM2020, code available at https://github.com/MILVLG/mmnas/

2604.02870 2026-04-06 cs.CV 72%

Token Warping Helps MLLMs Look from Nearby Viewpoints

令牌扭曲帮助多模态大语言模型从近处视角观察

Phillip Y. Lee, Chanho Park, Mingue Park, Seungwoo Yoo, Juil Koo, Minhyuk Sung

机构 * KAIST（韩国科学技术院）

专题命中 VLM训练与架构：visual reasoning(abstract)；multimodal large language model(abstract)；分类 cs.CV；MLLM(comments)

AI总结本文研究令牌扭曲如何帮助多模态大语言模型从近处视角理解场景，发现反向令牌扭曲在视角变化中更稳定且能更好保持语义一致性。

Comments CVPR 2026, Project Page: https://token-warping-mllm.github.io

2505.11121 2025-05-19 cs.CV 72%

Redundancy-Aware Pretraining of Vision-Language Foundation Models in Remote Sensing

Mathis Jürgen Adler, Leonard Hackel, Gencer Sumbul, Begüm Demir

机构 * TU Berlin（柏林技术大学）； BIFOLD（BIFOLD机构）； EPFL（苏黎世联邦理工学院）

专题命中 VLM训练与架构：VLM(abstract,comments)；vision-language model(abstract)；分类 cs.CV

Comments Accepted at IEEE International Geoscience and Remote Sensing Symposium (IGARSS) 2025. Our code is available at https://git.tu-berlin.de/rsim/redundacy-aware-rs-vlm

2508.10016 2026-05-25 cs.CL 71%

Training-Free Multimodal Large Language Model Orchestration

免训练的多模态大语言模型编排

Tianyu Xie, Yuexiao Ma, Yuhang Wu, Wang Chen, Jiayi Ji, Tat-Seng Chua, Xiawu Zheng, Rongrong Ji

机构 * Media Analytics and Computing Lab, Xiamen University, Xiamen, China（厦门大学媒体分析与计算实验室）； Institute of Artificial Intelligence, Xiamen University, Xiamen, China（厦门大学人工智能研究院）； School of Informatics, Xiamen University, Xiamen, China（厦门大学信息学院）； Peng Cheng Laboratory, Shenzhen, China（鹏城实验室）； Department of Computer Science, National University of Singapore, Singapore（新加坡国立大学计算机科学系）

专题命中 VLM训练与架构：multimodal large language model(title)

AI总结提出一种免训练编排框架LLM Orchestration，通过LLM控制器、文本中心跨模态记忆和统一交互层集成现成模态专家，实现低开销、可扩展的多模态输入输出系统。

Journal ref ICML 2026

2511.08126 2026-03-26 cs.CL 71%

Quantification and object perception in Multimodal Large Language Models and human linguistic cognition

多模态大语言模型和人类语言认知中的量化与物体感知

Raquel Montero, Natalia Moskvina, Paolo Morosi, Tamara Serrano, Elena Pagliarini, Evelina Leivada

机构 * Universitat Autònoma de Barcelona（巴塞罗那自治大学）； Institució Catalana de Recerca i Estudis Avançats (ICREA)（加泰罗尼亚高级科研与研究机构（ICREA））

专题命中 VLM训练与架构：multimodal large language model(title)

AI总结本文研究多模态大语言模型和人类在量化中的关键特征，探讨其在模型架构中的编码差异及语言影响。

2603.09627 2026-03-11 eess.AS 71%

Speech-Omni-Lite: Portable Speech Interfaces for Vision-Language Models

Speech-Omni-Lite: 用于视觉-语言模型的便携式语音接口

Dehua Tao, Xuan Luo, Daxin Tan, Kai Chen, Lanqing Hong, Jing Li, Ruifeng Xu, Xiao Chen

专题命中 VLM训练与架构：vision-language model(title)

AI总结 Speech-Omni-Lite通过轻量模块扩展视觉-语言模型，实现低成本语音理解和生成，同时保持原有性能，实验显示其在少量语音数据下表现优异。

2603.02250 2026-03-04 cs.SD eess.AS 71%

SGPA: Spectrogram-Guided Phonetic Alignment for Feasible Shapley Value Explanations in Multimodal Large Language Models

SGPA: 基于频谱的语音对齐用于多模态大语言模型中可行的谢普利值解释

Paweł Pozorski, Jakub Muszyński, Maria Ganzha

机构 * Warsaw University of Technology（华沙技术大学）

专题命中 VLM训练与架构：multimodal large language model(title)

AI总结 SGPA通过结合连接主义时间分类和频谱边界细化，实现了多模态大语言模型中可行的音频解释，显著减少了模型评估次数并保持了全局轮廓。

Comments Submitted for admission in Interspeech 2026 conference

2507.00236 2025-11-21 cs.RO 71%

Sim2Real Diffusion: Leveraging Foundation Vision Language Models for Adaptive Automated Driving

Sim2Real Diffusion：利用基础视觉语言模型实现自适应自动驾驶

Chinmay Vilas Samak, Tanmay Vilas Samak, Bing Li, Venkat Krovi

机构 * Department of Automotive Engineering, Clemson University International Center for Automotive Research (CU-ICAR)（汽车工程系，克莱姆森大学国际汽车研究中心（CU-ICAR））

专题命中 VLM训练与架构：vision language model(title)

AI总结本文提出Sim2Real Diffusion框架，利用基础视觉语言模型实现自动驾驶的跨领域适应，通过条件潜在扩散提升sim2real转换性能。

Comments Accepted in IEEE Robotics and Automation Letters (RA-L)

Journal ref IEEE Robotics and Automation Letters, vol. 11, no. 1, pp. 177-184, Jan. 2026

2408.11878 2025-06-10 cs.CL cs.CE q-fin.CP 71%

Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications

Jimin Huang, Mengxi Xiao, Dong Li, Zihao Jiang, Yuzhe Yang, Yifei Zhang, Lingfei Qian, Yan Wang, Xueqing Peng, Yang Ren, Ruoyu Xiang, Zhengyu Chen, Xiao Zhang, Yueru He, Weiguang Han, Shunian Chen, Lihang Shen, Daniel Kim, Yangyang Yu, Yupeng Cao, Zhiyang Deng, Haohang Li, Duanyu Feng, Yongfu Dai, VijayaSai Somasundaram, Peng Lu, Guojun Xiong, Zhiwei Liu, Zheheng Luo, Zhiyuan Yao, Ruey-Ling Weng, Meikang Qiu, Kaleb E Smith, Honghai Yu, Yanzhao Lai, Min Peng, Jian-Yun Nie, Jordan W. Suchow, Xiao-Yang Liu, Benyou Wang, Alejandro Lopez-Lira, Qianqian Xie, Sophia Ananiadou, Junichi Tsujii

机构 * The Fin AI（Fin AI）； Wuhan University（武汉大学）； Columbia University（哥伦比亚大学）； The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））； Nanjing University（南京大学）； Rensselaer Polytechnic Institute（罗格斯理工学院）； The University of Manchester（曼彻斯特大学）； Stevens Institute of Technology（史泰斯理工学院）； National University of Singapore（新加坡国立大学）； University of Florida（佛罗里达大学）； University of Montreal（蒙特利尔大学）； Yale University（耶鲁大学）； New York University（纽约大学）； Harvard University（哈佛大学）； NVIDIA（英伟达）； Artificial Intelligence Research Centre（人工智能研究中心）； Archimedes/Athena Research Centre（Archimedes/Athena研究中心）

专题命中 VLM训练与架构：multimodal large language model(title)

Comments 33 pages, 13 figures

2505.23830 2025-06-02 cs.CL 71%

EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Models

Linglin Jing, Yuting Gao, Zhigang Wang, Wang Lan, Yiwen Tang, Wenhai Wang, Kaipeng Zhang, Qingpei Guo

机构 * Shanghai AI Laboratory（上海人工智能实验室）； Ant Group（蚂蚁集团）

专题命中 VLM训练与架构：multimodal large language model(title)

2503.07111 2025-03-12 cs.RO cs.CL 71%

PoseLess: Depth-Free Vision-to-Joint Control via Direct Image Mapping with VLM

Alan Dao, Dinh Bach Vu, Tuan Le Duc Anh, Bui Quang Huy

专题命中 VLM训练与架构：VLM(title)

2407.02235 2025-03-10 cs.CL 71%

Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation

Cheng-Yi Li, Kao-Jung Chang, Cheng-Fu Yang, Hsin-Yu Wu, Wenting Chen, Hritik Bansal, Ling Chen, Yi-Ping Yang, Yu-Chun Chen, Shih-Pin Chen, Jiing-Feng Lirng, Kai-Wei Chang, Shih-Hwa Chiou

专题命中 VLM训练与架构：multimodal large language model(title)

Comments 6 figures, 5 supplementary figures, 8 supplementary tables

Journal ref Nature Communications 16, 2258 (2025)

2403.11399 2024-08-05 cs.CL 71%

X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment

Dongjae Shin, Hyeonseok Lim, Inho Won, Changsu Choi, Minjun Kim, Seungwoo Song, Hangyeol Yoo, Sangmin Kim, Kyungtae Lim

专题命中 VLM训练与架构：LLaVA(title)

Journal ref https://aclanthology.org/2024.findings-naacl.158

2109.07301 2021-09-16 cs.CL 71%

What Vision-Language Models `See' when they See Scenes

Michele Cafagna, Kees van Deemter, Albert Gatt

专题命中 VLM训练与架构：vision-language model(title)

1. VLM训练与架构 5095 篇

Multimodal LLMs Struggle with Basic Visual Network Analysis: a VNA Benchmark

OLIVE: Object Level In-Context Visual Embeddings

VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation

UniTable: Towards a Unified Framework for Table Recognition via Self-Supervised Pretraining

Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts

UNIAA: A Unified Multi-modal Image Aesthetic Assessment Baseline and Benchmark

GenEARL: A Training-Free Generative Framework for Multimodal Event Argument Role Labeling

SemGrasp: Semantic Grasp Generation via Language Aligned Discretization

SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining

Accountable Textual-Visual Chat Learns to Reject Human Instructions in Image Re-creation

SyCoCa: Symmetrizing Contrastive Captioners with Attentive Masking for Multimodal Alignment

Text Descriptions are Compressive and Invariant Representations for Visual Learning

Resolving References in Visually-Grounded Dialogue via Text Generation

Language Reward Modulation for Pretraining Reinforcement Learning

MIMIC-IT: Multi-Modal In-Context Instruction Tuning

Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment

Deep Multimodal Neural Architecture Search

Token Warping Helps MLLMs Look from Nearby Viewpoints

Redundancy-Aware Pretraining of Vision-Language Foundation Models in Remote Sensing

Training-Free Multimodal Large Language Model Orchestration

Quantification and object perception in Multimodal Large Language Models and human linguistic cognition

Speech-Omni-Lite: Portable Speech Interfaces for Vision-Language Models

SGPA: Spectrogram-Guided Phonetic Alignment for Feasible Shapley Value Explanations in Multimodal Large Language Models

Sim2Real Diffusion: Leveraging Foundation Vision Language Models for Adaptive Automated Driving

Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications

EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Models

PoseLess: Depth-Free Vision-to-Joint Control via Direct Image Mapping with VLM

Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generation

X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment

What Vision-Language Models `See' when they See Scenes