视觉大模型 / VLM

URECA: Unique Region Caption Anything

Sangbeom Lim, Junwan Kim, Heeji Yoon, Jaewoo Jung, Seungryong Kim

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments Project page: https://cvlab-kaist.github.io/URECA Code: https://github.com/cvlab-kaist/URECA

Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spots

Erfan Shayegani, G M Shahariar, Sara Abdali, Lei Yu, Nael Abu-Ghazaleh, Yue Dong

专题命中 VLM训练与架构 ：vision language model(abstract)；分类 cs.AI、cs.LG

RoboAct-CLIP: Video-Driven Pre-training of Atomic Action Understanding for Robotics

Zhiyuan Zhang, Yuxin He, Yong Sun, Junyu Shi, Lijiang Liu, Qiang Nie

专题命中 VLM训练与架构 ：visual language model(abstract)；分类 cs.AI、cs.LG

Comments IROS 2025

FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs

Mothilal Asokan, Kebin Wu, Fatima Albreiki

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Efficient LLaMA-3.2-Vision by Trimming Cross-attended Visual Features

Jewon Lee, Ki-Ung Song, Seungmin Yang, Donguk Lim, Jaeyeon Kim, Wooksu Shin, Bo-Kyeong Kim, Yong Jae Lee, Tae-Ho Kim

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments accepted at CVPR 2025 Workshop on ELVM

Enhancing Domain Adaptation through Prompt Gradient Alignment

Hoang Phan, Lam Tran, Quyen Tran, Trung Le

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted to NeurIPS 2024

BiPVL-Seg: Bidirectional Progressive Vision-Language Fusion with Global-Local Alignment for Medical Image Segmentation

Rafi Ibn Sultan, Hui Zhu, Chengyin Li, Dongxiao Zhu

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

LongViTU: Instruction Tuning for Long-Form Video Understanding

Rujie Wu, Xiaojian Ma, Hai Ci, Yue Fan, Yuxuan Wang, Haozhe Zhao, Qing Li, Yizhou Wang

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.LG

From Trial to Triumph: Advancing Long Video Understanding via Visual Context Sample Scaling and Self-reward Alignment

Yucheng Suo, Fan Ma, Linchao Zhu, Tianyi Wang, Fengyun Rao, Yi Yang

专题命中 VLM训练与架构 ：MLLM(abstract)；分类 cs.CV、cs.AI

Oasis: One Image is All You Need for Multimodal Instruction Data Synthesis

Letian Zhang, Quan Cui, Bingchen Zhao, Cheng Yang

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.AI

Data Augmentation in Earth Observation: A Diffusion Model Approach

Tiago Sousa, Benoît Ries, Nicolas Guelfi

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments 25 pages, 12 figures

Journal ref Information 2025, 16, 81

Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal Representations

Jeonghyeon Kim, Sangheum Hwang

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments CVPR 2025

Beyond Training: Dynamic Token Merging for Zero-Shot Video Understanding

Yiming Zhang, Zhuokai Zhao, Zhaorun Chen, Zenghui Ding, Xianjun Yang, Yining Sun

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.LG

Comments Code is available at https://github.com/Jam1ezhang/DYTO

JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation

Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai yu, Liang Zhao, Yisong Wang, Jiaying Liu, Chong Ruan

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted by CVPR 2025

Wolf: Dense Video Captioning with a World Summarization Framework

Boyi Li, Ligeng Zhu, Ran Tian, Shuhan Tan, Yuxiao Chen, Yao Lu, Yin Cui, Sushant Veer, Max Ehrlich, Jonah Philion, Xinshuo Weng, Fuzhao Xue, Linxi Fan, Yuke Zhu, Jan Kautz, Andrew Tao, Ming-Yu Liu, Sanja Fidler, Boris Ivanovic, Trevor Darrell, Jitendra Malik, Song Han, Marco Pavone

专题命中 VLM训练与架构 ：vision language model(abstract)；分类 cs.CV、cs.LG

Prompt Injection Attacks on Large Language Models in Oncology

Jan Clusmann, Dyke Ferber, Isabella C. Wiest, Carolin V. Schneider, Titus J. Brinker, Sebastian Foersch, Daniel Truhn, Jakob N. Kather

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.AI、cs.LG

Comments 57 Pages, 5 Figures

CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrieval

Yifan Xu, Xinhao Li, Yichun Yang, Desen Meng, Rui Huang, Limin Wang

专题命中 VLM训练与架构 ：MLLM(abstract)；分类 cs.CV、cs.LG

TacticExpert: Spatial-Temporal Graph Language Model for Basketball Tactics

Xu Lingrui, Liu Mandi, Zhang Lei

专题命中 VLM训练与架构 ：grounding(abstract)；分类 cs.AI、cs.LG

A Multimodal Benchmark Dataset and Model for Crop Disease Diagnosis

Xiang Liu, Zhaoxiang Liu, Huan Hu, Zezhou Chen, Kohou Wang, Kai Wang, Shiguo Lian

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ECCV 2024 (14 pages, 8 figures)

Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models

Yongxin Guo, Zhenglin Cheng, Xiaoying Tang, Zhaopeng Tu, Tao Lin

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.AI、cs.LG

Comments ICLR 2025

VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generation

Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, Song Han, Yao Lu

专题命中 VLM训练与架构 ：visual language model(abstract)；分类 cs.CV、cs.LG

Comments Code: https://github.com/mit-han-lab/vila-u. The first two authors contributed equally to this work

TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models

Makoto Shing, Kou Misaki, Han Bao, Sho Yokoi, Takuya Akiba

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.AI、cs.LG

Comments To appear at the 13th International Conference on Learning Representations (ICLR 2025) as a Spotlight presentation

SAN: Hypothesizing Long-Term Synaptic Development and Neural Engram Mechanism in Scalable Model's Parameter-Efficient Fine-Tuning

Gaole Dai, Chun-Kai Fan, Yiming Tang, Zhi Zhang, Yuan Zhang, Yulu Gan, Qizhe Zhang, Cheng-Ching Tseng, Shanghang Zhang, Tiejun Huang

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.AI、cs.LG

Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning

Weitai Kang, Haifeng Huang, Yuzhang Shang, Mubarak Shah, Yan Yan

专题命中 VLM训练与架构 ：grounding(abstract)；分类 cs.CV、cs.AI

Comments 8 pages

MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding

Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, Linda Ruth Petzold, Stephen D. Wilson, Woosang Lim, William Yang Wang

专题命中 VLM训练与架构 ：vision language model(abstract)；分类 cs.CV、cs.AI

Comments Code and data are available at https://github.com/Leezekun/MMSci

Vision-Based Generic Potential Function for Policy Alignment in Multi-Agent Reinforcement Learning

Hao Ma, Shijie Wang, Zhiqiang Pu, Siyao Zhao, Xiaolin Ai

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.AI、cs.LG

Intuitive physics understanding emerges from self-supervised pretraining on natural videos

Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, Yann LeCun

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments 24 pages,14 figures, 5 tables

COAT: Compressing Optimizer states and Activation for Memory-Efficient FP8 Training

Haocheng Xi, Han Cai, Ligeng Zhu, Yao Lu, Kurt Keutzer, Jianfei Chen, Song Han

专题命中 VLM训练与架构 ：vision language model(abstract)；分类 cs.AI、cs.LG

Comments Accepted by ICLR 2025. 22 pages. 9 Figures. 13 Tables

mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Data

Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, Zhicheng Dou

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

Information Theoretic Text-to-Image Alignment

Chao Wang, Giulio Franzese, Alessandro Finamore, Massimo Gallo, Pietro Michiardi

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments to appear at ICLR25