视觉大模型 / VLM

Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion

Jiuhai Chen, Jianwei Yang, Haiping Wu, Dianqi Li, Jianfeng Gao, Tianyi Zhou, Bin Xiao

专题命中 VLM训练与架构 ：vision-language model(title)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models

Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, Aniruddha Kembhavi

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.LG

Comments Updated with ablations and more technical details

Personalized Multimodal Large Language Models: A Survey

Junda Wu, Hanjia Lyu, Yu Xia, Zhehao Zhang, Joe Barrow, Ishita Kumar, Mehrnoosh Mirtaheri, Hongjie Chen, Ryan A. Rossi, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, Jiuxiang Gu, Nesreen K. Ahmed, Yu Wang, Xiang Chen, Hanieh Deilamsalehy, Namyong Park, Sungchul Kim, Huanrui Yang, Subrata Mitra, Zhengmian Hu, Nedim Lipka, Dang Nguyen, Yue Zhao, Jiebo Luo, Julian McAuley

专题命中 VLM训练与架构 ：multimodal large language model(title,abstract)；分类 cs.CV、cs.AI

FoPru: Focal Pruning for Efficient Large Vision-Language Models

Lei Jiang, Weizhe Huang, Tongxuan Liu, Yuting Zeng, Jing Li, Lechao Cheng, Xiaohua Xu

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Comments 11 pages, 7 figures

Towards Calibrated Robust Fine-Tuning of Vision-Language Models

Changdae Oh, Hyesu Lim, Mijoo Kim, Dongyoon Han, Sangdoo Yun, Jaegul Choo, Alexander Hauptmann, Zhi-Qi Cheng, Kyungwoo Song

专题命中 VLM训练与架构 ：vision-language model(title)；vision language model(abstract)；分类 cs.CV、cs.AI

Comments NeurIPS 2024 (a short version was presented at the NeurIPS 2023 Workshop on Distribution Shifts); Major modification of (v7): Fixing the x-axis of Figure 3 and Pearson correlation, accordingly

POINTS: Improving Your Vision-language Model with Affordable Strategies

Yuan Liu, Zhongyin Zhao, Ziyuan Zhuang, Le Tian, Xiao Zhou, Jie Zhou

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Comments v2

Frustratingly Easy Test-Time Adaptation of Vision-Language Models

Matteo Farina, Gianni Franchi, Giovanni Iacca, Massimiliano Mancini, Elisa Ricci

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Comments Camera-ready version for NeurIPS 2024

Calibrated Self-Rewarding Vision Language Models

Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, Huaxiu Yao

专题命中 VLM训练与架构 ：vision language model(title)；vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Added some experiments and charts, and redrew some figures V4

LLaVA-VSD: Large Language-and-Vision Assistant for Visual Spatial Description

Yizhang Jin, Jian Li, Jiangning Zhang, Jianlong Hu, Zhenye Gan, Xin Tan, Yong Liu, Yabiao Wang, Chengjie Wang, Lizhuang Ma

专题命中 VLM训练与架构 ：LLaVA(title,abstract)；分类 cs.CV、cs.AI

Comments We have discovered a significant error in the paper that affects the main conclusions. To ensure the accuracy of our research, we have decided to withdraw this paper and will resubmit it after making the necessary corrections

No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models

Angéline Pouget, Lucas Beyer, Emanuele Bugliarello, Xiao Wang, Andreas Peter Steiner, Xiaohua Zhai, Ibrahim Alabdulmohsin

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Comments 17 pages, 5 figures, 4 tables. 38th Conference on Neural Information Processing Systems (NeurIPS 2024)

Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, Junyang Lin

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Comments Code is available at https://github.com/QwenLM/Qwen2-VL. arXiv admin note: text overlap with arXiv:2408.15262 by other authors

Vision-Language Models are Strong Noisy Label Detectors

Tong Wei, Hao-Tian Li, Chun-Shu Li, Jiang-Xin Shi, Yu-Feng Li, Min-Ling Zhang

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.LG

Comments Accepted at NeurIPS 2024

A Survey of Multimodal Large Language Model from A Data-centric Perspective

Tianyi Bai, Hao Liang, Binwang Wan, Yanran Xu, Xi Li, Shiyu Li, Ling Yang, Bozhou Li, Yifan Wang, Bin Cui, Ping Huang, Jiulong Shan, Conghui He, Binhang Yuan, Wentao Zhang

专题命中 VLM训练与架构 ：multimodal large language model(title,abstract)；分类 cs.CV、cs.AI

Pre-Trained Vision-Language Models as Partial Annotators

Qian-Wei Wang, Yuqiu Xie, Letian Zhang, Zimo Liu, Shu-Tao Xia

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback

Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, Zackory Erickson

专题命中 VLM训练与架构 ：VLM(title,abstract)；分类 cs.AI、cs.LG

Comments ICML 2024

Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models

Shimin Chen, Yitian Yuan, Shaoxiang Chen, Zequn Jie, Lin Ma

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

The Neglected Tails in Vision-Language Models

Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caverlee, Shu Kong

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.LG

Comments Project Page: https://shubhamprshr27.github.io/neglected-tails-of-vlms/

Cross-Modal Adapter: Parameter-Efficient Transfer Learning Approach for Vision-Language Models

Juncheng Yang, Zuchao Li, Shuai Xie, Weiping Zhu, Wei Yu, Shijun Li

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.LG

Comments This paper is accepted to ICME 2024

DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback

Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, Ajay Divakaran

专题命中 VLM训练与架构 ：vision-language model(title)；vision language model(abstract)；分类 cs.CV、cs.LG

Comments CVPR 2024. The feedback datasets are released at: https://huggingface.co/datasets/YangyiYY/LVLM_NLF

LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images

Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, Gao Huang

专题命中 VLM训练与架构 ：LLaVA(title,abstract)；分类 cs.CV、cs.AI

Comments Preprint

Visual Data-Type Understanding does not emerge from Scaling Vision-Language Models

Vishaal Udandarao, Max F. Burg, Samuel Albanie, Matthias Bethge

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.LG

Semantic-Aware Frame-Event Fusion based Pattern Recognition via Large Vision-Language Models

Dong Li, Jiandong Jin, Yuhao Zhang, Yanlin Zhong, Yaoyang Wu, Lan Chen, Xiao Wang, Bin Luo

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Comments In Peer Review

Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare

Junling Liu, Ziming Wang, Qichen Ye, Dading Chong, Peilin Zhou, Yining Hua

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Class Incremental Learning with Pre-trained Vision-Language Models

Xialei Liu, Xusheng Cao, Haori Lu, Jia-wen Xiao, Andrew D. Bagdanov, Ming-Ming Cheng

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.LG

GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph

Xin Li, Dongze Lian, Zhihe Lu, Jiawang Bai, Zhibo Chen, Xinchao Wang

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted by NeurIPS 2023. The manuscript will be further revised based on the reviews

InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, Steven Hoi

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.LG

Comments preprint

PuMer: Pruning and Merging Tokens for Efficient Vision Language Models

Qingqing Cao, Bhargavi Paranjape, Hannaneh Hajishirzi

专题命中 VLM训练与架构 ：vision language model(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted to ACL 2023 Main Conference

You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model

Shengkun Tang, Yaqing Wang, Zhenglun Kong, Tianchi Zhang, Yao Li, Caiwen Ding, Yanzhi Wang, Yi Liang, Dongkuan Xu

专题命中 VLM训练与架构 ：vision language model(title)；visual language model(abstract)；分类 cs.CV、cs.LG

Prompt Tuning based Adapter for Vision-Language Model Adaption

Jingchen Sun, Jiayu Qin, Zihao Lin, Changyou Chen

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

A Multi-level Alignment Training Scheme for Video-and-Language Grounding

Yubo Zhang, Feiyang Niu, Qing Ping, Govind Thattai

专题命中 VLM训练与架构 ：grounding(title,abstract)；分类 cs.CV、cs.LG

Comments Accepted at ICDM 2022 FOMO-VL workshop

Journal ref 2022 IEEE International Conference on Data Mining Workshops (ICDMW), Orlando, FL, USA, 2022, pp. 958-966