视觉大模型 / VLM

MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer

Yanghao Li, Rui Qian, Bowen Pan, Haotian Zhang, Haoshuo Huang, Bowen Zhang, Jialing Tong, Haoxuan You, Xianzhi Du, Zhe Gan, Hyunjik Kim, Chao Jia, Zhenbang Wang, Yinfei Yang, Mingfei Gao, Zi-Yi Dou, Wenze Hu, Chang Gao, Dongxu Li, Philipp Dufter, Zirui Wang, Guoli Yin, Zhengdong Zhang, Chen Chen, Yang Zhao, Ruoming Pang, Zhifeng Chen

机构 * Apple（苹果公司） 纠错

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.LG

Discrete Diffusion in Large Language and Multimodal Models: A Survey

Runpeng Yu, Qi Li, Xinchao Wang

专题命中 VLM训练与架构 ：MLLM(abstract)；分类 cs.AI、cs.LG

DPDEdit: Detail-Preserved Diffusion Models for Multimodal Fashion Image Editing

Xiaolong Wang, Zhi-Qi Cheng, Jue Wang, Xiaojiang Peng

机构 * Shenzhen Technology University（深圳科技大学） ； University of Washington（华盛顿大学） ； Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院） 纠错

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments 13 pages,12 figures

Recurrence Meets Transformers for Universal Multimodal Retrieval

Davide Caffagni, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

机构 * Department of Education and Humanities, University of Modena and Reggio Emilia（教育与人文学院, Modena and Reggio Emilia大学） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Image Embedding Sampling Method for Diverse Captioning

Sania Waheed, Na Min An

机构 * University of Southhampton（南安普顿大学） ； KAIST（韩国科学技术院） 纠错

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.CV、cs.AI

Comments 17 pages, 5 figures, 9 tables

GAEA: A Geolocation Aware Conversational Assistant

Ron Campos, Ashmal Vayani, Parth Parag Kulkarni, Rohit Gupta, Aizan Zafar, Aritra Dutta, Mubarak Shah

机构 * Center for Research in Computer Vision, University of Central Florida（计算机视觉研究中心，中央佛罗里达大学） 纠错

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.LG

Comments The dataset and code used in this submission is available at: https://ucf-crcv.github.io/GAEA/

Fusion to Enhance: Fusion Visual Encoder to Enhance Multimodal Language Model

Yifei She, Huangxuan Wu

机构 * Beijing University of Posts and Telecommunications（北京邮电大学） 纠错

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

SpecVLM: Enhancing Speculative Decoding of Video LLMs via Verifier-Guided Token Pruning

Yicheng Ji, Jun Zhang, Heming Xia, Jinpeng Chen, Lidan Shou, Gang Chen, Huan Li

机构 * The State Key Laboratory of Blockchain and Data Security（区块链与数据安全国家重点实验室） ； Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院） ； Department of Computing, The Hong Kong Polytechnic University（香港理工大学计算机系） ； School of Computer Science, Beijing University of Posts and Telecommunications（北京邮电大学计算机学院） 纠错

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.AI

Comments Accepted at EMNLP 2025 Main

Real-Time Intuitive AI Drawing System for Collaboration: Enhancing Human Creativity through Formal and Contextual Intent Integration

Jookyung Song, Mookyoung Kang, Nojun Kwak

机构 * SNU（首尔国立大学） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments 6 pages, 4 figures, NeurIPS Creative AI Track 2025

VISA: Group-wise Visual Token Selection and Aggregation via Graph Summarization for Efficient MLLMs Inference

Pengfei Jiang, Hanjun Li, Linglan Zhao, Fei Chao, Ke Yan, Shouhong Ding, Rongrong Ji

机构 * Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（多媒体可信感知与高效计算重点实验室，教育部，厦门大学） ； Tencent Youtu Lab（腾讯优图实验室） 纠错

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ACMMM 2025

$Δ$-AttnMask: Attention-Guided Masked Hidden States for Efficient Data Selection and Augmentation

Jucheng Hu, Suorong Yang, Dongzhan Zhou

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions

Moran Yanuka, Assaf Ben Kish, Yonatan Bitton, Idan Szpektor, Raja Giryes

机构 * Tel Aviv University（特拉维夫大学） ； Google Research（谷歌研究） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted to NAACL 2025

Journal ref Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics, Human Language Technologies, Long Papers, pp. 10497-10518

Large-vocabulary forensic pathological analyses via prototypical cross-modal contrastive learning

Chen Shen, Chunfeng Lian, Wanqing Zhang, Fan Wang, Jianhua Zhang, Shuanliang Fan, Xin Wei, Gongji Wang, Kehan Li, Hongshu Mu, Hao Wu, Xinggong Liang, Jianhua Ma, Zhenyuan Wang

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.CV、cs.AI

Comments 28 pages, 6 figures, under review

Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents

Han Lin, Jaemin Cho, Amir Zadeh, Chuan Li, Mohit Bansal

机构 * UNC Chapel Hill（北卡罗来纳大学教堂山分校） ； Lambda 纠错

专题命中 VLM训练与架构 ：MLLM(abstract)；分类 cs.CV、cs.AI

Comments Project Page: https://bifrost-1.github.io

TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation

Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K. Du, Zehuan Yuan, Xinglong Wu

机构 * ByteDance（字节跳动） 纠错

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.AI

Comments CVPR 2025; Code and models: https://github.com/ByteVisionLab/TokenFlow

True Multimodal In-Context Learning Needs Attention to the Visual Context

Shuo Chen, Jianzhe Liu, Zhen Han, Yan Xia, Daniel Cremers, Philip Torr, Volker Tresp, Jindong Gu

机构 * LMU Munich（慕尼黑大学） ； Technical University of Munich（慕尼黑技术大学） ； Siemens AG（西门子股份公司） ； University of Science and Technology of China（中国科学技术大学） ； Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心） ； Konrad Zuse School of Excellence in Reliable AI (relAI)（Konrad Zuse可靠性人工智能卓越学院） ； University of Oxford（牛津大学） 纠错

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted to COLM 2025

CLIPTime: Time-Aware Multimodal Representation Learning from Images and Text

Anju Rani, Daniel Ortiz-Arroyo, Petar Durdevic

机构 * Department of Energy Technology（能源技术系） ； Aalborg University（奥尔堡大学） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments 11 pages, 8 figures

Decouple before Align: Visual Disentanglement Enhances Prompt Tuning

Fei Zhang, Tianfei Zhou, Jiangchao Yao, Ya Zhang, Ivor W. Tsang, Yanfeng Wang

机构 * Cooperative Medianet Innovation Center, Shanghai Jiao Tong University（合作中位网创新中心，上海交通大学） ； School of Artificial Intelligence, Shanghai Jiao Tong University（人工智能学院，上海交通大学） ； Shanghai Innovation Institute（上海创新研究院） ； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） ； Beijing Institute of Technology（北京理工大学） ； A*STAR Centre for Frontier AI Research（A*STAR前沿人工智能研究中心） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments 16 pages, Accepted at IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)

Towards Higher Effective Rank in Parameter-efficient Fine-tuning using Khatri--Rao Product

Paul Albert, Frederic Z. Zhang, Hemanth Saratchandran, Anton van den Hengel, Ehsan Abbasnejad

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments To appear in ICCV 2025

Enhancing Large Multimodal Models with Adaptive Sparsity and KV Cache Compression

Te Zhang, Yuheng Li, Junxiang Wang, Lujun Li

机构 * University of Michigan（密歇根大学） ； Johns Hopkins University（约翰霍普金斯大学） ； Central South university（中南大学） ； HKGAI 纠错

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.AI、cs.LG

Comments 6 pages

CopyJudge: Automated Copyright Infringement Identification and Mitigation in Text-to-Image Diffusion Models

Shunchang Liu, Zhuan Shi, Lingjuan Lyu, Yaochu Jin, Boi Faltings

机构 * EPFL（苏黎世联邦理工学院） ； Mila - Quebec AI Institute Mcgill University（蒙特利尔麦吉尔大学人工智能研究所） ； Westlake University（西湖大学） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ACM MM 2025

Efficient Learning for Product Attributes with Compact Multimodal Models

Mandar Kulkarni

机构 * Flipkart Data Science（Flipkart数据科学） 纠错

专题命中 VLM训练与架构 ：vision language model(abstract)；分类 cs.CV、cs.AI

Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding

StepFun, :, Bin Wang, Bojun Wang, Changyi Wan, Guanzhe Huang, Hanpeng Hu, Haonan Jia, Hao Nie, Mingliang Li, Nuo Chen, Siyu Chen, Song Yuan, Wuxun Xie, Xiaoniu Song, Xing Chen, Xingping Yang, Xuelin Zhang, Yanbo Yu, Yaoyu Wang, Yibo Zhu, Yimin Jiang, Yu Zhou, Yuanwei Lu, Houyi Li, Jingcheng Hu, Ka Man Lo, Ailin Huang, Binxing Jiao, Bo Li, Boyu Chen, Changxin Miao, Chang Lou, Chen Hu, Chen Xu, Chenfeng Yu, Chengyuan Yao, Daokuan Lv, Dapeng Shi, Deshan Sun, Ding Huang, Dingyuan Hu, Dongqing Pang, Enle Liu, Fajie Zhang, Fanqi Wan, Gulin Yan, Han Zhang, Han Zhou, Hanghao Wu, Hangyu Guo, Hanqi Chen, Hanshan Zhang, Hao Wu, Haocheng Zhang, Haolong Yan, Haoran Lv, Haoran Wei, Hebin Zhou, Heng Wang, Heng Wang, Hongxin Li, Hongyu Zhou, Hongyuan Wang, Huiyong Guo, Jia Wang, Jiahao Gong, Jialing Xie, Jian Zhou, Jianjian Sun, Jiaoren Wu, Jiaran Zhang, Jiayu Liu, Jie Cheng, Jie Luo, Jie Yan, Jie Yang, Jieyi Hou, Jinguang Zhang, Jinlan Cao, Jisheng Yin, Junfeng Liu, Junhao Huang, Junzhe Lin, Kaijun Tan, Kaixiang Li, Kang An, Kangheng Lin, Kenkun Liu, Lei Yang, Liang Zhao, Liangyu Chen, Lieyu Shi, Liguo Tan, Lin Lin, Lin Zhang, Lina Chen, Liwen Huang, Liying Shi, Longlong Gu, Mei Chen, Mengqiang Ren, Ming Li, Mingzhe Chen, Na Wang, Nan Wu, Qi Han, Qian Zhao, Qiang Zhang, Qianni Liu, Qiaohui Chen, Qiling Wu, Qinglin He, Qinyuan Tan, Qiufeng Wang, Qiuping Wu, Qiuyan Liang, Quan Sun, Rui Li, Ruihang Miao, Ruosi Wan, Ruyan Guo, Shangwu Zhong, Shaoliang Pang, Shengjie Fan, Shijie Shang, Shilei Jiang, Shiliang Yang, Shiming Hao, Shuli Gao, Siming Huang, Siqi Liu, Tiancheng Cao, Tianhao Cheng, Tianhao Peng, Wang You, Wei Ji, Wen Sun, Wenjin Deng, Wenqing He, Wenzhen Zheng, Xi Chen, Xiangwen Kong, Xianzhen Luo, Xiaobo Yang, Xiaojia Liu, Xiaoxiao Ren, Xin Han, Xin Li, Xin Wu, Xu Zhao, Yanan Wei, Yang Li, Yangguang Li, Yangshijie Xu, Yanming Xu, Yaqiang Shi, Yeqing Shen, Yi Yang, Yifei Yang, Yifeng Gong, Yihan Chen, Yijing Yang, Yinmin Zhang, Yizhuang Zhou, Yuanhao Ding, Yuantao Fan, Yuanzhen Yang, Yuchu Luo, Yue Peng, Yufan Lu, Yuhang Deng, Yuhe Yin, Yujie Liu, Yukun Chen, Yuling Zhao, Yun Mou, Yunlong Li, Yunzhou Ju, Yusheng Li, Yuxiang Yang, Yuxiang Zhang, Yuyang Chen, Zejia Weng, Zhe Xie, Zheng Ge, Zheng Gong, Zhenyi Lu, Zhewei Huang, Zhichao Chang, Zhiguo Huang, Zhirui Wang, Zidong Yang, Zili Wang, Ziqi Wang, Zixin Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, Xiangyu Zhang

机构 * StepFun Inc（StepFun公司） 纠错

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.AI、cs.LG

OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth?

Xuetian Chen, Yinghao Chen, Xinfeng Yuan, Zhuo Peng, Lu Chen, Yuekeng Li, Zhoujia Zhang, Yingqian Huang, Leyan Huang, Jiaqing Liang, Tianbao Xie, Zhiyong Wu, Qiushi Sun, Biqing Qi, Bowen Zhou

机构 * Fudan University（复旦大学） ； Shanghai AI Lab（上海人工智能实验室） ； Tsinghua University（清华大学） ； The University of Hong Kong（香港大学） 纠错

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.CV、cs.AI

Comments Work in progress

Steering into New Embedding Spaces: Analyzing Cross-Lingual Alignment Induced by Model Interventions in Multilingual Language Models

Anirudh Sundar, Sinead Williamson, Katherine Metcalf, Barry-John Theobald, Skyler Seto, Masha Fedzechkina

专题命中 VLM训练与架构 ：MLLM(abstract)；分类 cs.AI、cs.LG

Comments 34 pages

CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationale

Xiao Liang, Jiawei Hu, Di Wang, Zhi Ma, Lin Zhao, Ronghan Li, Bo Wan, Quan Wang

机构 * The Key Laboratory of Smart Human-Computer Interaction（智能人机交互关键实验室） ； Wearable Technology of Shaanxi Province, Xidian University, Xi'an, China（陕西 wearable 技术, 西安电子科技大学, 中国） ； Nanjing University of Science and Technology（南京理工大学） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Breach in the Shield: Unveiling the Vulnerabilities of Large Language Models

Runpeng Dai, Run Yang, Fan Zhou, Hongtu Zhu

机构 * University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校） ； BiliBili（哔哩哔哩） ； Shanghai University of Finance and Economics（上海财经大学） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.AI、cs.LG

Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation

Xi Zhang, Zaiqiao Meng, Jake Lever, Edmond S. L. Ho

机构 * School of Computing Science, University of Glasgow（计算科学学院，格拉斯哥大学） 纠错

专题命中 VLM训练与架构 ：visual language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted by BioNLP@ACL 2024

Journal ref Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, ACL 2024, pp. 624-634

Exploring Object Status Recognition for Recipe Progress Tracking in Non-Visual Cooking

Franklin Mingzhe Li, Kaitlyn Ng, Bin Zhu, Patrick Carrington

机构 * Carnegie Mellon University（卡内基梅隆大学） ； Singapore Management University（新加坡管理学院） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments ASSETS 2025

MGAA: Multi-Granular Adaptive Allocation fof Low-Rank Compression of LLMs

Guangyan Li, Yongqiang Tang, Wensheng Zhang

机构 * State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所） ； School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院） ； School of Computer Science and Cyber Engineering, Guangzhou University（广州大学计算机科学与网络工程学院） 纠错

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.AI、cs.LG

Comments 13 pages, 8 figures