视觉大模型 / VLM

Text as Image: Learning Transferable Adapter for Multi-Label Classification

Xuelin Zhu, Jiuxin Cao, Jian liu, Dongqi Tang, Furong Xu, Weijia Liu, Jiawei Ge, Bo Liu, Qingpei Guo, Tianyi Zhang

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV

SILC: Improving Vision Language Pretraining with Self-Distillation

Muhammad Ferjad Naeem, Yongqin Xian, Xiaohua Zhai, Lukas Hoyer, Luc Van Gool, Federico Tombari

专题命中 VLM训练与架构 ：visual question answering(abstract)；分类 cs.CV

VTimeLLM: Empower LLM to Grasp Video Moments

Bin Huang, Xin Wang, Hong Chen, Zihan Song, Wenwu Zhu

专题命中 VLM训练与架构 ：grounding(abstract)；分类 cs.CV

HKUST at SemEval-2023 Task 1: Visual Word Sense Disambiguation with Context Augmentation and Visual Assistance

Zhuohao Yin, Xin Huang

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV

To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning

Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, Yu-Gang Jiang

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV

Comments techical report; work in progress

ShareGPT4V: Improving Large Multi-Modal Models with Better Captions

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, Dahua Lin

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV

Comments Project: https://ShareGPT4V.github.io

CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning

Hongyu Hu, Jiyuan Zhang, Minyi Zhao, Zhenbang Sun

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV

Unlock the Power: Competitive Distillation for Multi-Modal Large Language Models

Xinwei Li, Li Lin, Shuai Wang, Chen Qian

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV

LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark

Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, Jing Shao, Wanli Ouyang

专题命中 VLM训练与架构 ：MLLM(abstract)；分类 cs.CV

Comments NeurIPS2023 camera ready ; 37 pages, 33 figures. Code available at https://github.com/OpenLAMM/LAMM ; Project page: https://openlamm.github.io/

PILL: Plug Into LLM with Adapter Expert and Attention Gate

Fangyuan Zhang, Tingting Liang, Zhengyuan Wu, Yuyu Yin

专题命中 VLM训练与架构 ：vision language model(abstract)；分类 cs.CV

CAD -- Contextual Multi-modal Alignment for Dynamic AVQA

Asmar Nadeem, Adrian Hilton, Robert Dawes, Graham Thomas, Armin Mustafa

专题命中 VLM训练与架构 ：visual question answering(abstract)；分类 cs.CV

Comments Accepted to IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2024

Tuning Multi-mode Token-level Prompt Alignment across Modalities

Dongsheng Wang, Miaoge Li, Xinyang Liu, MingSheng Xu, Bo Chen, Hanwang Zhang

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV

Comments In Proceedings of NeurIPS2023

MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, Mohamed Elhoseiny

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV

Comments Project Website: https://minigpt-4.github.io/; Code, Pretrained Model, and Dataset: https://github.com/Vision-CAIR/MiniGPT-4; Deyao Zhu and Jun Chen contributed equally to this work

TAP: Targeted Prompting for Task Adaptive Generation of Textual Training Instances for Visual Classification

M. Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Horst Possegger, Rogerio Feris, Horst Bischof

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.CV

Comments Code is available at: https://github.com/jmiemirza/TAP

ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoders

Shawn Xu, Lin Yang, Christopher Kelly, Marcin Sieniek, Timo Kohlberger, Martin Ma, Wei-Hung Weng, Atilla Kiraly, Sahar Kazemzadeh, Zakkai Melamed, Jungyeon Park, Patricia Strachan, Yun Liu, Chuck Lau, Preeti Singh, Christina Chen, Mozziyar Etemadi, Sreenivasa Raju Kalidindi, Yossi Matias, Katherine Chou, Greg S. Corrado, Shravya Shetty, Daniel Tse, Shruthi Prabhakara, Daniel Golden, Rory Pilgrim, Krish Eswaran, Andrew Sellergren

专题命中 VLM训练与架构 ：visual question answering(abstract)；分类 cs.CV

BDC-Adapter: Brownian Distance Covariance for Better Vision-Language Reasoning

Yi Zhang, Ce Zhang, Zihan Liao, Yushun Tang, Zhihai He

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV

Comments Accepted by BMVC 2023

Orthogonal Temporal Interpolation for Zero-Shot Video Recognition

Yan Zhu, Junbao Zhuo, Bin Ma, Jiajia Geng, Xiaoming Wei, Xiaolin Wei, Shuhui Wang

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV

Journal ref Proceedings of the 31st ACM International Conference on Multimedia (MM '23), October 29-November 3, 2023

3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment

Ziyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng, Siyuan Huang, Qing Li

专题命中 VLM训练与架构 ：grounding(abstract)；分类 cs.CV

GIST: Generating Image-Specific Text for Fine-grained Object Classification

Kathleen M. Lewis, Emily Mu, Adrian V. Dalca, John Guttag

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV

Comments The first two authors contributed equally to this work and are listed in alphabetical order

RegionBLIP: A Unified Multi-modal Pre-training Framework for Holistic and Regional Comprehension

Qiang Zhou, Chaohui Yu, Shaofeng Zhang, Sitong Wu, Zhibing Wang, Fan Wang

专题命中 VLM训练与架构 ：MLLM(abstract)；分类 cs.CV

ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning

Liang Zhao, En Yu, Zheng Ge, Jinrong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chunrui Han, Xiangyu Zhang

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV

Comments 15 pages, 8 figures

TaCA: Upgrading Your Visual Foundation Model with Task-agnostic Compatible Adapter

Binjie Zhang, Yixiao Ge, Xuyuan Xu, Ying Shan, Mike Zheng Shou

专题命中 VLM训练与架构 ：visual question answering(abstract)；分类 cs.CV

ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understanding

Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, Silvio Savarese

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV

Comments Accepted by CVPR 2023

VILA: Learning Image Aesthetics from User Comments with Vision-Language Pretraining

Junjie Ke, Keren Ye, Jiahui Yu, Yonghui Wu, Peyman Milanfar, Feng Yang

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV

Comments CVPR 2023, https://github.com/google-research/google-research/tree/master/vila

Few-Shot Learning with Visual Distribution Calibration and Cross-Modal Distribution Alignment

Runqi Wang, Hao Zheng, Xiaoyue Duan, Jianzhuang Liu, Yuning Lu, Tian Wang, Songcen Xu, Baochang Zhang

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV

Measuring Progress in Fine-grained Vision-and-Language Understanding

Emanuele Bugliarello, Laurent Sartran, Aishwarya Agrawal, Lisa Anne Hendricks, Aida Nematzadeh

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.CV

Comments ACL 2023

Simple Token-Level Confidence Improves Caption Correctness

Suzanne Petryk, Spencer Whitehead, Joseph E. Gonzalez, Trevor Darrell, Anna Rohrbach, Marcus Rohrbach

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV

Rethinking Benchmarks for Cross-modal Image-text Retrieval

Weijing Chen, Linli Yao, Qin Jin

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.CV

Comments Accepted to SIGIR2023

Movie Box Office Prediction With Self-Supervised and Visually Grounded Pretraining

Qin Chao, Eunsoo Kim, Boyang Li

专题命中 VLM训练与架构 ：grounding(abstract)；分类 cs.LG

Comments accepted by IEEE International Conference on Multimedia and Expo (2023)

CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition

Deepti Hegde, Jeya Maria Jose Valanarasu, Vishal M. Patel

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV

Comments Website: https://jeya-maria-jose.github.io/cg3d-web/