多模态大模型

IQAGPT: Image Quality Assessment with Vision-language and ChatGPT Models

Zhihao Chen, Bin Hu, Chuang Niu, Tao Chen, Yuxin Li, Hongming Shan, Ge Wang

机构 * ISTBI Fudan University（ISTBI 复旦大学） ； Huashan Hospital Fudan University（复旦大学华山医院） ； BME & CBIS Rensselaer Polytechnic Institute（生物医学工程与生物信息学研究中心罗切斯特理工学院） 纠错

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments 14 pages, 9 figures

Journal ref Visual Computing for Industry, Biomedicine, and Art, 7, 20, 2024

SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models

Zhenwei Tang, Difan Jiao, Blair Yang, Ashton Anderson

机构 * Department of Computer Science, University of Toronto（多伦多大学计算机科学系） ； Coolwei AI Lab（Coolwei人工智能实验室） 纠错

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments COLM 2025

SynthVLM: Towards High-Quality and Efficient Synthesis of Image-Caption Datasets for Vision-Language Models

Zheng Liu, Hao Liang, Bozhou Li, Wentao Xiong, Chong Chen, Conghui He, Wentao Zhang, Bin Cui

机构 * Peking University Beijing China ； Huawei Technologies Ltd. Beijing China ； Shanghai AI Laboratory Shanghai China ； Peking University Center for Machine Learning Research Beijing China ； Peking University School of Computer Science \& Key Lab of High Confidence Software Technologies (MOE) Beijing China ； Peking University ； Huawei Technologies Ltd. ； Shanghai AI Laboratory 纠错

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens

Haohan Zheng, Zhenguo Zhang

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Cure or Poison? Embedding Instructions Visually Alters Hallucination in Vision-Language Models

Zhaochen Wang, Yiwei Wang, Yujun Cai

机构 * The University of Queensland（昆士兰大学） ； University of California, Merced（加州大学默塞德分校） 纠错

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Work in progress

Meta CLIP 2: A Worldwide Scaling Recipe

Yung-Sung Chuang, Yang Li, Dong Wang, Ching-Feng Yeh, Kehan Lyu, Ramya Raghavendra, James Glass, Lifei Huang, Jason Weston, Luke Zettlemoyer, Xinlei Chen, Zhuang Liu, Saining Xie, Wen-tau Yih, Shang-Wen Li, Hu Xu

机构 * FAIR, Meta（FAIR与Meta公司） ； MIT（麻省理工学院） ； Princeton University（普林斯顿大学） ； New York University（纽约大学） 纠错

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments 10 pages

MR-CLIP: Efficient Metadata-Guided Learning of MRI Contrast Representations

Mehmet Yigit Avci, Pedro Borges, Paul Wright, Mehmet Yigitsoy, Sebastien Ourselin, Jorge Cardoso

机构 * School of Biomedical Engineering and Imaging Sciences, King’s College London, London, UK（生物医学工程与成像科学学院，伦敦国王学院，伦敦，英国） ； deepc GMBH, Munich, Germany（deepc 德国慕尼黑分公司） 纠错

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Spatial 3D-LLM: Exploring Spatial Awareness in 3D Vision-Language Models

Xiaoyan Wang, Zeju Li, Yifan Xu, Jiaxing Qi, Zhifei Yang, Ruifei Ma, Xiangde Liu, Chao Zhang

机构 * Beijing Digital Native Digital City Research Center（北京数字原生数字城市研究中心） ； The Chinese University of Hong Kong（香港中文大学） 纠错

专题命中 图文多模态 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ICME2025

Skywork-R1V3 Technical Report

Wei Shen, Jiangbo Pei, Yi Peng, Xuchen Song, Yang Liu, Jian Peng, Haofeng Sun, Yunzhuo Hao, Peiyu Wang, Jianhao Zhang, Yahui Zhou

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning

Pengkun Jiao, Bin Zhu, Jingjing Chen, Chong-Wah Ngo, Yu-Gang Jiang

机构 * Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University（上海智能信息处理关键实验室，复旦大学计算机学院） ； Shanghai Collaborative Innovation Center on Intelligent Visual Computing（上海智能视觉计算协同创新中心） ； Singapore Management University（新加坡管理大学） 纠错

专题命中 图文多模态 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments ICCV 2025

A Survey on Vision-Language-Action Models for Autonomous Driving

Sicong Jiang, Zilin Huang, Kangan Qian, Ziang Luo, Tianze Zhu, Yang Zhong, Yihong Tang, Menglin Kong, Yunlong Wang, Siwen Jiao, Hao Ye, Zihao Sheng, Xin Zhao, Tuopu Wen, Zheng Fu, Sikai Chen, Kun Jiang, Diange Yang, Seongjin Choi, Lijun Sun

机构 * McGill University（麦吉尔大学） ； Tsinghua University（清华大学） ； Xiaomi Corporation（小米公司） ； University of Wisconsin–Madison（威斯康星大学麦迪逊分校） ； University of Minnesota–Twin Cities（明尼苏达大学双城分校） ； State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University（智能绿色车辆与移动国家重点实验室，清华大学） 纠错

专题命中 图文多模态 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Generalizing vision-language models to novel domains: A comprehensive survey

Xinyao Li, Jingjing Li, Fengling Li, Lei Zhu, Yang Yang, Heng Tao Shen

机构 * University of Electronic Science and Technology of China（电子科技大学） ； University of Technology Sydney（技术学院） ； Tongji University（同济大学） 纠错

专题命中 图文多模态 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Unifying Biomedical Vision-Language Expertise: Towards a Generalist Foundation Model via Multi-CLIP Knowledge Distillation

Shansong Wang, Zhecheng Jin, Mingzhe Hu, Mojtaba Safari, Feng Zhao, Chih-Wei Chang, Richard LJ Qiu, Justin Roper, David S. Yu, Xiaofeng Yang

机构 * Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine（放射肿瘤科，Winship癌症研究所，埃默里大学医学院） ； Department of Biomedical Engineering, College of Engineering, Georgia Institute of Technology（生物医学工程系，工程学院，佐治亚理工学院） ； Department of Computer Science and Mathematics, Laney Graduate School, Emory University（计算机科学与数学系，兰尼研究生院，埃默里大学） ； School of Electrical and Computer Engineering, College of Engineering, Georgia Institute of Technology（电气与计算机工程系，工程学院，佐治亚理工学院） 纠错

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Robo2VLM: Visual Question Answering from Large-Scale In-the-Wild Robot Manipulation Datasets

Kaiyuan Chen, Shuangyu Xie, Zehan Ma, Pannag R Sanketi, Ken Goldberg

专题命中 图文多模态 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CL、cs.AI

SensorLM: Learning the Language of Wearable Sensors

Yuwei Zhang, Kumar Ayush, Siyuan Qiao, A. Ali Heydari, Girish Narayanswamy, Maxwell A. Xu, Ahmed A. Metwally, Shawn Xu, Jake Garrison, Xuhai Xu, Tim Althoff, Yun Liu, Pushmeet Kohli, Jiening Zhan, Mark Malhotra, Shwetak Patel, Cecilia Mascolo, Xin Liu, Daniel McDuff, Yuzhe Yang

机构 * Google DeepMind（谷歌DeepMind） ； Google Research（谷歌研究） ； University of Cambridge（剑桥大学） 纠错

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI

Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks

Mengzhao Jia, Wenhao Yu, Kaixin Ma, Tianqing Fang, Zhihan Zhang, Siru Ouyang, Hongming Zhang, Dong Yu, Meng Jiang

机构 * University of Notre Dame（诺丁汉大学） ； Tencent AI Seattle Lab（腾讯AI西雅图实验室） ； UIUC（伊利诺伊大学香槟分校） 纠错

专题命中 图文多模态 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.CL

Comments Our code is available at https://github.com/tencent-ailab/Leopard

Curriculum Learning with Quality-Driven Data Selection

Biao Wu, Ling Chen

机构 * Australian Artificial Intelligence Institute（澳大利亚人工智能研究所） 纠错

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CL、cs.AI

CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified Multiplet Upcycling

Jihai Zhang, Xiaoye Qu, Tong Zhu, Yu Cheng

机构 * First Author Affiliation（第一作者机构） ； Second Author Affiliation（第二作者机构） 纠错

专题命中 图文多模态 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

FG-CLIP: Fine-Grained Visual and Textual Alignment

Chunyu Xie, Bin Wang, Fanjing Kong, Jincheng Li, Dawei Liang, Gengshen Zhang, Dawei Leng, Yuhui Yin

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments Accepted at ICML 2025

Low-hallucination Synthetic Captions for Large-Scale Vision-Language Model Pre-training

Xinsong Zhang, Yarong Zeng, Xinting Huang, Hu Hu, Runquan Xie, Han Hu, Zhanhui Kang

机构 * Tencent（腾讯） 纠错

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Data-Driven Calibration of Prediction Sets in Large Vision-Language Models Based on Inductive Conformal Prediction

Yuanchang Ye, Weiyan Wen

专题命中 图文多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI

Comments Accepted by ICIPCA 2025

PathVLM-R1: A Reinforcement Learning-Driven Reasoning Model for Pathology Visual-Language Tasks

Jianyu Wu, Hao Yang, Xinhua Zeng, Guibing He, Zhiyu Chen, Zihui Li, Xiaochuan Zhang, Yangyang Ma, Run Fang, Yang Liu

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Post-pre-training for Modality Alignment in Vision-Language Foundation Models

Shin'ya Yamaguchi, Dewei Feng, Sekitoshi Kanai, Kazuki Adachi, Daiki Chijiwa

专题命中 图文多模态 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments Accepted to CVPR 2025; Code: https://github.com/yshinya6/clip-refine

CameraBench: Benchmarking Visual Reasoning in MLLMs via Photography

I-Sheng Fang, Jun-Cheng Chen

专题命中 图文多模态 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.CL

ReferGPT: Towards Zero-Shot Referring Multi-Object Tracking

Tzoulio Chamiti, Leandro Di Bella, Adrian Munteanu, Nikos Deligiannis

专题命中 图文多模态 ：multi-modal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments Accepted CVPR 2025 Workshop on Distillation of Foundation Models for Autonomous Driving

BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature

Alejandro Lozano, Min Woo Sun, James Burgess, Liangyu Chen, Jeffrey J Nirschl, Jeffrey Gu, Ivan Lopez, Josiah Aklilu, Austin Wolfgang Katzer, Collin Chiu, Anita Rau, Xiaohan Wang, Yuhui Zhang, Alfred Seunghoon Song, Robert Tibshirani, Serena Yeung-Levy

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference

Hao Yin, Guangzong Si, Zilei Wang

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Treble Counterfactual VLMs: A Causal Approach to Hallucination

Shawn Li, Jiashu Qu, Yuxiao Zhou, Yuehan Qin, Tiankai Yang, Yue Zhao

专题命中 图文多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

PiSA: A Self-Augmented Data Engine and Training Strategy for 3D Understanding with Large Models

Zilu Guo, Hongbin Lin, Zhihao Yuan, Chaoda Zheng, Pengshuo Qiu, Dongzhi Jiang, Renrui Zhang, Chun-Mei Feng, Zhen Li

专题命中 图文多模态 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments Technical Report

DiffCLIP: Differential Attention Meets CLIP

Hasan Abed Al Kader Hammoud, Bernard Ghanem

专题命中 图文多模态 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments Under review