视觉大模型 / VLM

Realistic Unsupervised CLIP Fine-tuning with Universal Entropy Optimization

Jian Liang, Lijun Sheng, Zhengbo Wang, Ran He, Tieniu Tan

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments ICML 2024 Highlight

Towards a text-based quantitative and explainable histopathology image analysis

Anh Tien Nguyen, Trinh Thi Le Vuong, Jin Tae Kwak

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments MICCAI 2024 - Early acceptance (Top 11%)

Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision

Orr Zohar, Xiaohan Wang, Yonatan Bitton, Idan Szpektor, Serena Yeung-Levy

专题命中 VLM训练与架构 ：vision language model(abstract)；分类 cs.CV、cs.AI

Comments Project page: https://orrzohar.github.io/projects/video-star/

μ-Bench: A Vision-Language Benchmark for Microscopy Understanding

Alejandro Lozano, Jeffrey Nirschl, James Burgess, Sanket Rajan Gupte, Yuhui Zhang, Alyssa Unell, Serena Yeung-Levy

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

NExT-GPT: Any-to-Any Multimodal LLM

Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, Tat-Seng Chua

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.AI、cs.LG

Comments ICML 2024 (Oral)

PRESTO: Progressive Pretraining Enhances Synthetic Chemistry Outcomes

He Cao, Yanjun Shao, Zhiyuan Liu, Zijing Liu, Xiangru Tang, Yuan Yao, Yu Li

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.AI、cs.LG

How Well Do Multi-modal LLMs Interpret CT Scans? An Auto-Evaluation Framework for Analyses

Qingqing Zhu, Benjamin Hou, Tejas S. Mathai, Pritam Mukherjee, Qiao Jin, Xiuying Chen, Zhizheng Wang, Ruida Cheng, Ronald M. Summers, Zhiyong Lu

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.AI

MINT: a Multi-modal Image and Narrative Text Dubbing Dataset for Foley Audio Content Planning and Generation

Ruibo Fu, Shuchen Shi, Hongming Guo, Tao Wang, Chunyu Qiang, Zhengqi Wen, Jianhua Tao, Xin Qi, Yi Lu, Xiaopeng Wang, Zhiyong Wang, Yukun Liu, Xuefei Liu, Shuai Zhang, Guanjun Li

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.AI

Direct Preference Optimization for Suppressing Hallucinated Prior Exams in Radiology Report Generation

Oishi Banerjee, Hong-Yu Zhou, Subathra Adithan, Stephen Kwak, Kay Wu, Pranav Rajpurkar

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Added acknowledgemnts

SViTT-Ego: A Sparse Video-Text Transformer for Egocentric Video

Hector A. Valdez, Kyle Min, Subarna Tripathi

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment

Sajid Javed, Arif Mahmood, Iyyakutti Iyappan Ganapathi, Fayaz Ali Dharejo, Naoufel Werghi, Mohammed Bennamoun

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Synth$^2$: Boosting Visual-Language Models with Synthetic Captions and Image Embeddings

Sahand Sharifzadeh, Christos Kaplanis, Shreya Pathak, Dharshan Kumaran, Anastasija Ilic, Jovana Mitrovic, Charles Blundell, Andrea Banino

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.CV、cs.AI

Comments 9 pages, 6 figures

Prompt-based Visual Alignment for Zero-shot Policy Transfer

Haihan Gao, Rui Zhang, Qi Yi, Hantao Yao, Haochen Li, Jiaming Guo, Shaohui Peng, Yunkai Gao, QiCheng Wang, Xing Hu, Yuanbo Wen, Zihao Zhang, Zidong Du, Ling Li, Qi Guo, Yunji Chen

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.CV、cs.AI

Comments This paper has been accepted by ICML2024

Exploiting LMM-based knowledge for image classification tasks

Maria Tzelepi, Vasileios Mezaris

专题命中 VLM训练与架构 ：vision language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted for publication, 25th Int. Conf. on Engineering Applications of Neural Networks (EANN/EAAAI 2024), Corfu, Greece, June 2024. This is the "submitted manuscript"

Multimodal Deep Learning for Low-Resource Settings: A Vector Embedding Alignment Approach for Healthcare Applications

David Restrepo, Chenwei Wu, Sebastián Andrés Cajas, Luis Filipe Nakayama, Leo Anthony Celi, Diego M López

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.AI、cs.LG

WIDIn: Wording Image for Domain-Invariant Representation in Single-Source Domain Generalization

Jiawei Ma, Yulei Niu, Shiyuan Huang, Guangxing Han, Shih-Fu Chang

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Explaining Multi-modal Large Language Models by Analyzing their Vision Perception

Loris Giulivi, Giacomo Boracchi

专题命中 VLM训练与架构 ：MLLM(abstract)；分类 cs.CV、cs.AI

Comments Submitted at BMVC 2024

LDEdit: Towards Generalized Text Guided Image Manipulation via Latent Diffusion Models

Paramanand Chandramouli, Kanchana Vaishnavi Gandikota

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted BMVC 2022

MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, Yinfei Yang

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.LG

CFIR: Fast and Effective Long-Text To Image Retrieval for Large Corpora

Zijun Long, Xuri Ge, Richard Mccreadie, Joemon Jose

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification

Qinying Liu, Wei Wu, Kecheng Zheng, Zhan Tong, Jiawei Liu, Yu Liu, Wei Chen, Zilei Wang, Yujun Shen

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

VidLA: Video-Language Alignment at Scale

Mamshad Nayeem Rizve, Fan Fei, Jayakrishnan Unnikrishnan, Son Tran, Benjamin Z. Yao, Belinda Zeng, Mubarak Shah, Trishul Chilimbi

专题命中 VLM训练与架构 ：grounding(abstract)；分类 cs.CV、cs.LG

Comments Accepted to CVPR 2024

Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuning

Cristina Menghini, Andrew Delworth, Stephen H. Bach

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

KG-TREAT: Pre-training for Treatment Effect Estimation by Synergizing Patient Data with Knowledge Graphs

Ruoqi Liu, Lingfei Wu, Ping Zhang

专题命中 VLM训练与架构 ：grounding(abstract)；分类 cs.AI、cs.LG

Comments AAAI 2024 Main Track

Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion

Xuantong Liu, Tianyang Hu, Wenjia Wang, Kenji Kawaguchi, Yuan Yao

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.AI、cs.LG

LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptions

Songhao Han, Le Zhuo, Yue Liao, Si Liu

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Emergent Communication Pretraining for Few-Shot Machine Translation

Yaoyiran Li, Edoardo M. Ponti, Ivan Vulić, Anna Korhonen

专题命中 VLM训练与架构 ：grounding(abstract)；分类 cs.AI、cs.LG

Journal ref Proceedings of the 28th International Conference on Computational Linguistics (2020) pages 4716-4731

M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretraining

Qingpei Guo, Furong Xu, Hanxiao Zhang, Wang Ren, Ziping Ma, Lin Ju, Jian Wang, Jingdong Chen, Ming Yang

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.CV、cs.AI

Vision-Language Foundation Models as Effective Robot Imitators

Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, Hang Li, Tao Kong

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.AI、cs.LG

Comments Fix typos. Project page: https://roboflamingo.github.io

Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Loss

Jordan Shipard, Arnold Wiliem, Kien Nguyen Thanh, Wei Xiang, Clinton Fookes

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments 15 pages