多模态大模型 - arXivDaily 专题

2505.20029 2025-05-27 q-bio.NC cs.AI cs.LG 83%

Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)

Subba Reddy Oota, Akshett Jindal, Ishani Mondal, Khushbu Pahwa, Satya Sai Srinath Namburi, Manish Shrivastava, Maneesh Singh, Bapi S. Raju, Manish Gupta

机构 * Technische Universität Berlin（柏林技术大学）； IIIT Hyderabad（海得拉巴国家理工学院）； Univ of Maryland（马里兰大学）； Rice Univ（Rice 大学）； Univ of Wisconsin - Madison（威斯康星大学麦迪逊分校）； Spector Inc（Spector 公司）； Microsoft（微软公司）

专题命中图文多模态：multimodal(title,abstract)；MLLM(abstract)；分类 cs.AI

Comments 30 pages, 22 figures, The Thirteenth International Conference on Learning Representations, ICLR-2025, Singapore. https://openreview.net/pdf?id=xkgfLXZ4e0

Journal ref ICLR-2025, Singapore

2505.19812 2025-05-27 cs.CV 83%

Efficient Multi-modal Long Context Learning for Training-free Adaptation

Zehong Ma, Shiliang Zhang, Longhui Wei, Qi Tian

专题命中图文多模态：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments Accepted to ICML2025

2505.06356 2025-05-13 cs.CV 83%

Understanding and Mitigating Toxicity in Image-Text Pretraining Datasets: A Case Study on LLaVA

Karthik Reddy Kanjula, Surya Guthikonda, Nahid Alam, Shayekh Bin Islam

机构 * Cohere for AI Community（Cohere for AI社区）； Cisco Meraki ； Indiana University Bloomington（印第安纳大学布卢明顿分校）； Bangladesh University of Engineering and Technology（孟加拉工程与技术大学）

专题命中图文多模态：image-text(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments Accepted at ReGenAI CVPR2025 Workshop as Oral

2505.04601 2025-05-08 cs.CV 83%

OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning

Xianhang Li, Yanqing Liu, Haoqin Tu, Hongru Zhu, Cihang Xie

机构 * University of California, Santa Cruz（加州大学圣克鲁兹分校）

专题命中图文多模态：multimodal(title,abstract)；multimodal foundation model(abstract)；分类 cs.CV

2505.03703 2025-05-07 cs.CV cs.LG 83%

Fill the Gap: Quantifying and Reducing the Modality Gap in Image-Text Representation Learning

François Role, Sébastien Meyer, Victor Amblard

机构 * Université Paris-Cité（巴黎-cite大学）； Pôle d’Expertise de la Régulation Numérique (PEReN)（数字监管专家中心）

专题命中图文多模态：image-text(title,abstract)；multimodal(abstract)；分类 cs.CV

2505.02013 2025-05-06 cs.CV 83%

MLLM-Enhanced Face Forgery Detection: A Vision-Language Fusion Solution

Siran Peng, Zipei Wang, Li Gao, Xiangyu Zhu, Tianshuo Zhang, Ajian Liu, Haoyuan Zhang, Zhen Lei

机构 * CASIA ； UCAS ； CMCC ； CAIR, HKISI, CAS

专题命中图文多模态：MLLM(title,abstract)；multimodal(abstract)；分类 cs.CV

2412.04332 2025-04-14 cs.CV 83%

Liquid: Language Models are Scalable and Unified Multi-modal Generators

Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, Xiang Bai

专题命中图文多模态：multi-modal(title)；multimodal(abstract)；MLLM(abstract)；分类 cs.CV

Comments Technical report. Project page: https://foundationvision.github.io/Liquid/

2404.18930 2025-04-03 cs.CV 83%

Hallucination of Multimodal Large Language Models: A Survey

Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, Mike Zheng Shou

专题命中图文多模态：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments 228 references

2503.10437 2025-04-02 cs.CV 83%

4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models

Wanhua Li, Renping Zhou, Jiawei Zhou, Yingwei Song, Johannes Herter, Minghan Qin, Gao Huang, Hanspeter Pfister

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments CVPR 2025. Project Page: https://4d-langsplat.github.io

2503.20502 2025-04-01 cs.CV 83%

MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuning

Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Jiayi Ji, Jie Lou, Debing Zhang, Rongrong Ji

专题命中图文多模态：MLLM(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Tech Report

2411.14522 2025-03-28 cs.CV 83%

GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI

Tianbin Li, Yanzhou Su, Wei Li, Bin Fu, Zhe Chen, Ziyan Huang, Guoan Wang, Chenglong Ma, Ying Chen, Ming Hu, Yanjun Li, Pengcheng Chen, Xiaowei Hu, Zhongying Deng, Yuanfeng Ji, Jin Ye, Yu Qiao, Junjun He

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

2410.10491 2025-03-21 cs.CV 83%

TWIST & SCOUT: Grounding Multimodal LLM-Experts by Forget-Free Tuning

Aritra Bhowmik, Mohammad Mahdi Derakhshani, Dennis Koelma, Yuki M. Asano, Martin R. Oswald, Cees G. M. Snoek

专题命中图文多模态：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

2501.06828 2025-03-14 cs.CV 83%

GeoPix: Multi-Modal Large Language Model for Pixel-level Image Understanding in Remote Sensing

Ruizhe Ou, Yuan Hu, Fan Zhang, Jiaxin Chen, Yu Liu

专题命中图文多模态：multi-modal(title,abstract)；MLLM(abstract)；分类 cs.CV

2503.06446 2025-03-11 cs.CV 83%

M$^3$amba: CLIP-driven Mamba Model for Multi-modal Remote Sensing Classification

Mingxiang Cao, Weiying Xie, Xin Zhang, Jiaqing Zhang, Kai Jiang, Jie Lei, Yunsong Li

专题命中图文多模态：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2503.06106 2025-03-11 cs.CV 83%

Vision-aware Multimodal Prompt Tuning for Uploadable Multi-source Few-shot Domain Adaptation

Kuanghong Liu, Jin Wang, Kangjian He, Dan Xu, Xuejie Zhang

专题命中图文多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by AAAI 2025

2503.04353 2025-03-07 cs.CV 83%

ObjMST: An Object-Focused Multimodal Style Transfer Framework

Chanda Grover Kamra, Indra Deep Mastan, Debayan Gupta

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments 8 pages, 8 Figures, 3 Tables

Journal ref Pattern Recognition Letters, Special Issues, 2024

2503.03651 2025-03-06 cs.CV 83%

DoraCycle: Domain-Oriented Adaptation of Unified Generative Model in Multimodal Cycles

Rui Zhao, Weijia Mao, Mike Zheng Shou

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments CVPR 2025

2406.05127 2025-02-27 cs.CV 83%

Towards Semantic Equivalence of Tokenization in Multimodal LLM

Shengqiong Wu, Hao Fei, Xiangtai Li, Jiayi Ji, Hanwang Zhang, Tat-Seng Chua, Shuicheng Yan

专题命中图文多模态：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments ICLR-2025. The project page: https://chocowu.github.io/SeTok-web/

2502.17516 2025-02-26 cs.LG cs.AI 83%

A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models

Zihao Lin, Samyadeep Basu, Mohammad Beigi, Varun Manjunatha, Ryan A. Rossi, Zichao Wang, Yufan Zhou, Sriram Balasubramanian, Arman Zarei, Keivan Rezaei, Ying Shen, Barry Menglong Yao, Zhiyang Xu, Qin Liu, Yuxiang Zhang, Yan Sun, Shilong Liu, Li Shen, Hongxuan Li, Soheil Feizi, Lifu Huang

专题命中图文多模态：multi-modal(title)；multimodal(abstract)；multimodal foundation model(abstract)；分类 cs.AI

Comments 30 pages, 4 Figures, 10 Tables

2405.13459 2025-02-18 cs.CV 83%

Adapting Multi-modal Large Language Model to Concept Drift From Pre-training Onwards

Xiaoyu Yang, Jie Lu, En Yu

专题命中图文多模态：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments ICLR 2025 Poster

2502.09906 2025-02-17 cs.CV 83%

Insect-Foundation: A Foundation Model and Large Multimodal Dataset for Vision-Language Insect Understanding

Thanh-Dat Truong, Hoang-Quan Nguyen, Xuan-Bac Nguyen, Ashley Dowling, Xin Li, Khoa Luu

专题命中图文多模态：multimodal(title,abstract)；multimodal foundation model(abstract)；分类 cs.CV

2501.18006 2025-01-31 cs.LG cs.AI cs.CR 83%

Topological Signatures of Adversaries in Multimodal Alignments

Minh Vu, Geigh Zollicoffer, Huy Mai, Ben Nebgen, Boian Alexandrov, Manish Bhattarai

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.AI

2501.15269 2025-01-28 cs.LG cs.CR cs.CV 83%

Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink

Yining Wang, Mi Zhang, Junjie Sun, Chenyue Wang, Min Yang, Hui Xue, Jialing Tao, Ranjie Duan, Jiexi Liu

专题命中图文多模态：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments USENIX Security 2025

2501.01720 2025-01-27 cs.CV 83%

Interpretable Face Anti-Spoofing: Enhancing Generalization with Multimodal Large Language Models

Guosheng Zhang, Keyao Wang, Haixiao Yue, Ajian Liu, Gang Zhang, Kun Yao, Errui Ding, Jingdong Wang

专题命中图文多模态：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments Accepted to AAAI2025(Oral)

2501.11469 2025-01-22 cs.CV cs.LG 83%

MASS: Overcoming Language Bias in Image-Text Matching

Jiwan Chung, Seungwon Lim, Sangkyu Lee, Youngjae Yu

专题命中图文多模态：image-text(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments AAAI 2025

2411.10950 2025-01-14 cs.CL 83%

Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering

Zeping Yu, Sophia Ananiadou

专题命中图文多模态：multimodal(title)；multi-modal(abstract)；MLLM(abstract)；分类 cs.CL

Comments preprint

2311.07594 2025-01-09 cs.CL cs.AI cs.CV cs.MM 83%

How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model

Shezheng Song, Xiaopeng Li, Shasha Li, Shan Zhao, Jie Yu, Jun Ma, Xiaoguang Mao, Weimin Zhang

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by TKDE

2409.05840 2025-01-03 cs.CL 83%

MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct

Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Minzheng Wang, Pengpeng Zeng, Lianli Gao, Heng Tao Shen, Yunshui Li, Xiaobo Xia, Fei Huang, Jingkuan Song, Yongbin Li

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CL

2406.08394 2025-01-03 cs.CV 83%

VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks

Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, Ping Luo, Yu Qiao, Jifeng Dai

专题命中图文多模态：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments 44 pages

2412.20927 2024-12-31 cs.CV 83%

Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering

Junxiao Xue, Quan Deng, Fei Yu, Yanhao Wang, Jun Wang, Yuehua Li

专题命中图文多模态：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments 6 pages, 3 figures, under review

1. 图文多模态 4682 篇

Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)

Efficient Multi-modal Long Context Learning for Training-free Adaptation

Understanding and Mitigating Toxicity in Image-Text Pretraining Datasets: A Case Study on LLaVA

OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning

Fill the Gap: Quantifying and Reducing the Modality Gap in Image-Text Representation Learning

MLLM-Enhanced Face Forgery Detection: A Vision-Language Fusion Solution

Liquid: Language Models are Scalable and Unified Multi-modal Generators

Hallucination of Multimodal Large Language Models: A Survey

4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models

MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuning

GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AI

TWIST & SCOUT: Grounding Multimodal LLM-Experts by Forget-Free Tuning

GeoPix: Multi-Modal Large Language Model for Pixel-level Image Understanding in Remote Sensing

M$^3$amba: CLIP-driven Mamba Model for Multi-modal Remote Sensing Classification

Vision-aware Multimodal Prompt Tuning for Uploadable Multi-source Few-shot Domain Adaptation

ObjMST: An Object-Focused Multimodal Style Transfer Framework

DoraCycle: Domain-Oriented Adaptation of Unified Generative Model in Multimodal Cycles

Towards Semantic Equivalence of Tokenization in Multimodal LLM

A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models

Adapting Multi-modal Large Language Model to Concept Drift From Pre-training Onwards

Insect-Foundation: A Foundation Model and Large Multimodal Dataset for Vision-Language Insect Understanding

Topological Signatures of Adversaries in Multimodal Alignments

Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink

Interpretable Face Anti-Spoofing: Enhancing Generalization with Multimodal Large Language Models

MASS: Overcoming Language Bias in Image-Text Matching

Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering

How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Model

MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct

VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks

Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering