视觉大模型 / VLM - arXivDaily 专题

2409.12319 2025-03-10 cs.CV cs.MM cs.SD eess.AS 70%

Large Language Models are Strong Audio-Visual Speech Recognition Learners

Umberto Cappellazzo, Minsu Kim, Honglie Chen, Pingchuan Ma, Stavros Petridis, Daniele Falavigna, Alessio Brutti, Maja Pantic

专题命中 VLM训练与架构：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

Comments Accepted for publication at ICASSP 2025. The code and checkpoints are available here: https://github.com/umbertocappellazzo/Llama-AVSR

2503.04417 2025-03-07 cs.AI cs.MA 70%

From Idea to CAD: A Language Model-Driven Multi-Agent System for Collaborative Design

Felix Ocker, Stefan Menzel, Ahmed Sadik, Thiago Rios

专题命中 VLM训练与架构：vision language model(abstract)；VLM(abstract)；分类 cs.AI

Comments 11 pages, 3 figures

2502.20035 2025-02-28 cs.CV 70%

AsymLoRA: Harmonizing Data Conflicts and Commonalities in MLLMs

Xuyang Wei, Chunlin Tian, Li Li

专题命中 VLM训练与架构：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

2406.05127 2025-02-27 cs.CV 70%

Towards Semantic Equivalence of Tokenization in Multimodal LLM

Shengqiong Wu, Hao Fei, Xiangtai Li, Jiayi Ji, Hanwang Zhang, Tat-Seng Chua, Shuicheng Yan

专题命中 VLM训练与架构：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

Comments ICLR-2025. The project page: https://chocowu.github.io/SeTok-web/

2403.20271 2025-02-25 cs.CV 70%

Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want

Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shanghang Zhang, Hongsheng Li

专题命中 VLM训练与架构：LLaVA(abstract)；multimodal large language model(abstract)；分类 cs.CV

Comments 30 pages, 8 figures, 15 tables

2502.09057 2025-02-14 cs.CV 70%

Vision-Language In-Context Learning Driven Few-Shot Visual Inspection Model

Shiryu Ueno, Yoshikazu Hayashi, Shunsuke Nakatsuka, Yusei Yamada, Hiroaki Aizawa, Kunihito Kato

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments VISAPP 2025

2502.08391 2025-02-13 cs.CV 70%

ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification

Jiangbo Shi, Chen Li, Tieliang Gong, Yefeng Zheng, Huazhu Fu

专题命中 VLM训练与架构：vision language model(abstract)；VLM(abstract)；分类 cs.CV

Comments CVPR 2024 (Updated version with corrections for typos and errors.)

2502.07183 2025-02-13 cs.RO cs.CV 70%

Space-Aware Instruction Tuning: Dataset and Benchmark for Guide Dog Robots Assisting the Visually Impaired

ByungOk Han, Woo-han Yun, Beom-Su Seo, Jaehong Kim

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments ICRA 2025

2403.19622 2025-02-04 cs.RO cs.CV 70%

RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agents

Zeren Chen, Zhelun Shi, Xiaoya Lu, Lehan He, Sucheng Qian, Zhenfei Yin, Wanli Ouyang, Jing Shao, Yu Qiao, Cewu Lu, Lu Sheng

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments 18 pages, 11 figures, 7 tables. Accepted by NeurIPS 2024 Workshop

2408.12109 2025-02-03 cs.CV cs.CL 70%

RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data

Chenglong Wang, Yang Gan, Yifu Huo, Yongyu Mu, Murun Yang, Qiaozhi He, Tong Xiao, Chunliang Zhang, Tongran Liu, Quan Du, Di Yang, Jingbo Zhu

专题命中 VLM训练与架构：vision-language model(abstract)；LLaVA(abstract)；分类 cs.CV

Comments Accepted by AAAI 2025

2501.16629 2025-01-29 cs.CL cs.CV 70%

CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs

Jinlan Fu, Shenzhen Huangfu, Hao Fei, Xiaoyu Shen, Bryan Hooi, Xipeng Qiu, See-Kiong Ng

专题命中 VLM训练与架构：LLaVA(abstract)；multimodal large language model(abstract)；分类 cs.CV

Comments Accepted by ICLR 2025

2404.14047 2025-01-14 cs.LG 70%

An empirical study of LLaMA3 quantization: from LLMs to MLLMs

Wei Huang, Xingyu Zheng, Xudong Ma, Haotong Qin, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, Michele Magno

专题命中 VLM训练与架构：LLaVA(abstract)；MLLM(abstract)；分类 cs.LG

2408.16228 2025-01-09 cs.RO cs.LG 70%

Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitation

Vivek Myers, Bill Chunyuan Zheng, Oier Mees, Sergey Levine, Kuan Fang

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.LG

Comments 27 pages, 14 figures

Journal ref Conference on Robot Learning, 2024

2403.13352 2025-01-03 cs.CV 70%

AGFSync: Leveraging AI-Generated Feedback for Preference Optimization in Text-to-Image Generation

Jingkun An, Yinghao Zhu, Zongjian Li, Enshen Zhou, Haoran Feng, Xijie Huang, Bohua Chen, Yemin Shi, Chengwei Pan

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments Accepted by AAAI-2025

2412.16211 2024-12-24 cs.CV cs.CL cs.GR 70%

Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generation

Yiping Wang, Xuehai He, Kuan Wang, Luyao Ma, Jianwei Yang, Shuohang Wang, Simon Shaolei Du, Yelong Shen

专题命中 VLM训练与架构：vision-language model(abstract)；LLaVA(abstract)；分类 cs.CV

Comments benchmark paper, project page: https://ypwang61.github.io/project/StoryEval

2406.04295 2024-12-17 cs.CV 70%

Everything to the Synthetic: Diffusion-driven Test-time Adaptation via Synthetic-Domain Alignment

Jiayi Guo, Junhao Zhao, Chaoqun Du, Yulin Wang, Chunjiang Ge, Zanlin Ni, Shiji Song, Humphrey Shi, Gao Huang

专题命中 VLM训练与架构：LLaVA(abstract)；multimodal large language model(abstract)；分类 cs.CV

Comments GitHub: https://github.com/SHI-Labs/Diffusion-Driven-Test-Time-Adaptation-via-Synthetic-Domain-Alignment

2412.10372 2024-12-16 cs.CV 70%

UniMed-CLIP: Towards a Unified Image-Text Pretraining Paradigm for Diverse Medical Imaging Modalities

Muhammad Uzair Khattak, Shahina Kunhimon, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments Code, models and demo available at https://github.com/mbzuai-oryx/UniMed-CLIP

2412.09618 2024-12-13 cs.CV 70%

EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM

Zhuofan Zong, Dongzhi Jiang, Bingqi Ma, Guanglu Song, Hao Shao, Dazhong Shen, Yu Liu, Hongsheng Li

专题命中 VLM训练与架构：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

Comments Tech report

2412.09604 2024-12-13 cs.CV 70%

SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding

Hao Li, Changyao Tian, Jie Shao, Xizhou Zhu, Zhaokai Wang, Jinguo Zhu, Wenhan Dou, Xiaogang Wang, Hongsheng Li, Lewei Lu, Jifeng Dai

专题命中 VLM训练与架构：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

2412.07112 2024-12-11 cs.CV cs.CL 70%

Maya: An Instruction Finetuned Multilingual Multimodal Model

Nahid Alam, Karthik Reddy Kanjula, Surya Guthikonda, Timothy Chung, Bala Krishna S Vegesna, Abhipsha Das, Anthony Susevski, Ryan Sze-Yin Chan, S M Iftekhar Uddin, Shayekh Bin Islam, Roshan Santhosh, Snegha A, Drishti Sharma, Chen Liu, Isha Chaturvedi, Genta Indra Winata, Ashvanth. S, Snehanshu Mukherjee, Alham Fikri Aji

专题命中 VLM训练与架构：vision-language model(abstract)；LLaVA(abstract)；分类 cs.CV

2412.06673 2024-12-10 cs.CV 70%

ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance

Chunwei Wang, Guansong Lu, Junwei Yang, Runhui Huang, Jianhua Han, Lu Hou, Wei Zhang, Hang Xu

专题命中 VLM训练与架构：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

2412.03555 2024-12-05 cs.CV 70%

PaliGemma 2: A Family of Versatile VLMs for Transfer

Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, Siyang Qin, Reeve Ingle, Emanuele Bugliarello, Sahar Kazemzadeh, Thomas Mesnard, Ibrahim Alabdulmohsin, Lucas Beyer, Xiaohua Zhai

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

2412.02531 2024-12-04 cs.CV 70%

Multimodal Remote Sensing Scene Classification Using VLMs and Dual-Cross Attention Networks

Jinjin Cai, Kexin Meng, Baijian Yang, Gang Shao

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

2412.01725 2024-12-03 cs.CV 70%

Attacks on multimodal models

Viacheslav Iablochnikov, Alexander Rogachev

专题命中 VLM训练与架构：VLM(abstract)；LLaVA(abstract)；分类 cs.CV

Comments 19 pages, 13 figures, 3 tables

2412.01343 2024-12-03 cs.CV 70%

MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models

Xiaomin Li, Xu Jia, Qinghe Wang, Haiwen Diao, Mengmeng Ge, Pengxiang Li, You He, Huchuan Lu

专题命中 VLM训练与架构：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

Comments Accepted by ACM MM 2024, code will be released in https://github.com/XiaominLi1997/MoTrans

2407.11522 2024-12-03 cs.CV 70%

FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models

Pengxiang Li, Zhi Gao, Bofei Zhang, Tao Yuan, Yuwei Wu, Mehrtash Harandi, Yunde Jia, Song-Chun Zhu, Qing Li

专题命中 VLM训练与架构：vision language model(abstract)；LLaVA(abstract)；分类 cs.CV

Comments NeurIPS 2024 Dataset & Benchmark Track

2411.17837 2024-11-28 cs.CV 70%

OracleSage: Towards Unified Visual-Linguistic Understanding of Oracle Bone Scripts through Cross-Modal Knowledge Fusion

Hanqi Jiang, Yi Pan, Junhao Chen, Zhengliang Liu, Yifan Zhou, Peng Shu, Yiwei Li, Huaqin Zhao, Stephen Mihm, Lewis C Howe, Tianming Liu

专题命中 VLM训练与架构：vision-language model(abstract)；LLaVA(abstract)；分类 cs.CV

2411.16828 2024-11-27 cs.CV 70%

CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions

Yanqing Liu, Xianhang Li, Zeyu Wang, Bingchen Zhao, Cihang Xie

专题命中 VLM训练与架构：LLaVA(abstract)；MLLM(abstract)；分类 cs.CV

Comments 12 pages

2411.13909 2024-11-25 cs.CV 70%

Panther: Illuminate the Sight of Multimodal LLMs with Instruction-Guided Visual Prompts

Honglin Li, Yuting Gao, Chenglu Zhu, Jingdong Chen, Ming Yang, Lin Yang

专题命中 VLM训练与架构：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

2411.10161 2024-11-18 cs.CV 70%

SEAGULL: No-reference Image Quality Assessment for Regions of Interest via Vision-Language Instruction Tuning

Zewen Chen, Juan Wang, Wen Wang, Sunhan Xu, Hang Xiong, Yun Zeng, Jian Guo, Shuxun Wang, Chunfeng Yuan, Bing Li, Weiming Hu

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

1. VLM训练与架构 5084 篇

Large Language Models are Strong Audio-Visual Speech Recognition Learners

From Idea to CAD: A Language Model-Driven Multi-Agent System for Collaborative Design

AsymLoRA: Harmonizing Data Conflicts and Commonalities in MLLMs

Towards Semantic Equivalence of Tokenization in Multimodal LLM

Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want

Vision-Language In-Context Learning Driven Few-Shot Visual Inspection Model

ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification

Space-Aware Instruction Tuning: Dataset and Benchmark for Guide Dog Robots Assisting the Visually Impaired

RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agents

RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data

CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs

An empirical study of LLaMA3 quantization: from LLMs to MLLMs

Policy Adaptation via Language Optimization: Decomposing Tasks for Few-Shot Imitation

AGFSync: Leveraging AI-Generated Feedback for Preference Optimization in Text-to-Image Generation

Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generation

Everything to the Synthetic: Diffusion-driven Test-time Adaptation via Synthetic-Domain Alignment

UniMed-CLIP: Towards a Unified Image-Text Pretraining Paradigm for Diverse Medical Imaging Modalities

EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM

SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding

Maya: An Instruction Finetuned Multilingual Multimodal Model

ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance

PaliGemma 2: A Family of Versatile VLMs for Transfer

Multimodal Remote Sensing Scene Classification Using VLMs and Dual-Cross Attention Networks

Attacks on multimodal models

MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models

FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models

OracleSage: Towards Unified Visual-Linguistic Understanding of Oracle Bone Scripts through Cross-Modal Knowledge Fusion

CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions

Panther: Illuminate the Sight of Multimodal LLMs with Instruction-Guided Visual Prompts

SEAGULL: No-reference Image Quality Assessment for Regions of Interest via Vision-Language Instruction Tuning