视觉大模型 / VLM

Concept-skill Transferability-based Data Selection for Large Vision-Language Models

Jaewoo Lee, Boyang Li, Sung Ju Hwang

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.LG

Comments EMNLP 2024

Attention Prompting on Image for Large Vision-Language Models

Runpeng Yu, Weihao Yu, Xinchao Wang

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

Comments Website, see https://yu-rp.github.io/api-prompting

Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Reset

Andrew Goldberg, Kavish Kondap, Tianshuang Qiu, Zehan Ma, Letian Fu, Justin Kerr, Huang Huang, Kaiyuan Chen, Kuan Fang, Ken Goldberg

专题命中 VLM训练与架构 ：VLM(title,abstract)；vision language model(abstract)；分类 cs.AI、cs.LG

Comments 8 pages, 7 Figures

KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Data

Grace Tang, Swetha Rajkumar, Yifei Zhou, Homer Rich Walke, Sergey Levine, Kuan Fang

专题命中 VLM训练与架构 ：vision-language model(title)；vision language model(abstract)；VLM(abstract)；分类 cs.AI、cs.LG

Comments 8 pages, 7 figures

TG-LLaVA: Text Guided LLaVA via Learnable Latent Embeddings

Dawei Yan, Pengcheng Li, Yang Li, Hao Chen, Qingguo Chen, Weihua Luo, Wei Dong, Qingsen Yan, Haokui Zhang, Chunhua Shen

专题命中 VLM训练与架构 ：LLaVA(title,abstract)；vision-language model(abstract)；分类 cs.CV、cs.AI

Do Pre-trained Vision-Language Models Encode Object States?

Kaleb Newman, Shijie Wang, Yuan Zang, David Heffren, Chen Sun

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing

Zilun Zhang, Tiancheng Zhao, Yulong Guo, Jianwei Yin

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments RS5M dataset v5

Journal ref TGRS-2023-06174

An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models

Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, Baobao Chang

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

Comments Accepted to ECCV 2024 (Oral), code is released at https://github.com/pkunlp-icler/FastV,

Building and better understanding vision-language models: insights and future directions

Hugo Laurençon, Andrés Marafioti, Victor Sanh, Léo Tronchon

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning

Jiajie Li, Garrett Skinner, Gene Yang, Brian R Quaranto, Steven D Schwaitzberg, Peter C W Kim, Jinjun Xiong

专题命中 VLM训练与架构 ：LLaVA(title,abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

GP-VLS: A general-purpose vision language model for surgery

Samuel Schmidgall, Joseph Cho, Cyril Zakka, William Hiesinger

专题命中 VLM训练与架构 ：vision language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Reinforcement Learning Friendly Vision-Language Model for Minecraft

Haobin Jiang, Junpeng Yue, Hao Luo, Ziluo Ding, Zongqing Lu

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Comments ECCV 2024

Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Models

Ivan Kapelyukh, Yifei Ren, Ignacio Alzugaray, Edward Johns

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Comments ICRA 2024. Project webpage with robot videos: https://www.robot-learning.uk/dream2real

Noise-Tolerant Few-Shot Unsupervised Adapter for Vision-Language Models

Eman Ali, Muhammad Haris Khan

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；vision language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted at BMVC 2024

LLaVA-Read: Enhancing Reading Ability of Multimodal Language Models

Ruiyi Zhang, Yufan Zhou, Jian Chen, Jiuxiang Gu, Changyou Chen, Tong Sun

专题命中 VLM训练与架构 ：LLaVA(title,abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments NeurIPS 2024 Under Review

Vision Language Model-Empowered Contract Theory for AIGC Task Allocation in Teleoperation

Zijun Zhan, Yaxian Dong, Yuqing Hu, Shuai Li, Shaohua Cao, Zhu Han

专题命中 VLM训练与架构 ：vision language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments 11 pages, 10 figures

Rethinking Pruning for Vision-Language Models: Strategies for Effective Sparsity and Performance Restoration

Shwai He, Ang Li, Tianlong Chen

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity

Zhenlin Xu, Yi Zhu, Tiffany Deng, Abhay Mittal, Yanbei Chen, Manchen Wang, Paolo Favaro, Joseph Tighe, Davide Modolo

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments CVPR2024 MMFM workshop

Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?

Xiujun Li, Yujie Lu, Zhe Gan, Jianfeng Gao, William Yang Wang, Yejin Choi

专题命中 VLM训练与架构 ：multimodal large language model(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments Github: https://github.com/VIM-Bench/VIM_TOOL, Model and Data: https://huggingface.co/VIM-Bench

MLLMReID: Multimodal Large Language Model-based Person Re-identification

Shan Yang, Yongfei Zhang

专题命中 VLM训练与架构 ：multimodal large language model(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models

Jinhao Li, Haopeng Li, Sarah Erfani, Lei Feng, James Bailey, Feng Liu

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Comments 22 pages, 16 figures, published to ICML 2024

Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding

Xintong Wang, Jingheng Pan, Liang Ding, Chris Biemann

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

Comments Accepted to Findings of ACL 2024

Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models

Yue Zhang, Hehe Fan, Yi Yang

专题命中 VLM训练与架构 ：multimodal large language model(title,abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs

Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

专题命中 VLM训练与架构 ：LLaVA(title,abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments CVPR 2024 Workshop on What is Next in Multimodal Foundation Models

What matters when building vision-language models?

Hugo Laurençon, Léo Tronchon, Matthieu Cord, Victor Sanh

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

VG4D: Vision-Language Model Goes 4D Video Recognition

Zhichao Deng, Xiangtai Li, Xia Li, Yunhai Tong, Shen Zhao, Mengyuan Liu

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments ICRA 2024

On Speculative Decoding for Multimodal Large Language Models

Mukul Gagrani, Raghavv Goel, Wonseok Jeon, Junyoung Park, Mingu Lee, Christopher Lott

专题命中 VLM训练与架构 ：multimodal large language model(title,abstract)；LLaVA(abstract)；分类 cs.AI、cs.LG

Comments Accepted as a spotlight paper to ELVM workshop at CVPR 2024

Progressive Alignment with VLM-LLM Feature to Augment Defect Classification for the ASE Dataset

Chih-Chung Hsu, Chia-Ming Lee, Chun-Hung Sun, Kuang-Ming Wu

专题命中 VLM训练与架构 ：VLM(title,abstract)；vision-language model(abstract)；分类 cs.CV、cs.LG

Comments MULA 2024

Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models

Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, Jiaya Jia

专题命中 VLM训练与架构 ：vision language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments Code and models are available at https://github.com/dvlab-research/MiniGemini

UniCode: Learning a Unified Codebook for Multimodal Large Language Models

Sipeng Zheng, Bohan Zhou, Yicheng Feng, Ye Wang, Zongqing Lu

专题命中 VLM训练与架构 ：multimodal large language model(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments 14 pages, 2 figures, 11 tables