多模态大模型 - arXivDaily 专题

2501.17391 2025-10-01 cs.CV cs.AI cs.CL 85%

LFTR: Learning-Free Token Reduction for Multimodal Large Language Models

Zihui Zhao, Yingxin Li, Yang Li

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

2509.24298 2025-09-30 cs.HC cs.AI cs.CL cs.CY cs.MM 85%

Bridging the behavior-neural gap: A multimodal AI reveals the brain's geometry of emotion more accurately than human self-reports

Changde Du, Yizhuo Lu, Zhongyu Huang, Yi Sun, Zisen Zhou, Shaozheng Qin, Huiguang He

机构 * State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology, Institute of Automation, Chinese Academy of Sciences（脑认知与脑启发智能技术重点实验室，自动化研究所，中国科学院）； School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）； School of Future Technology, University of Chinese Academy of Sciences（未来技术学院，中国科学院大学）； State Key Laboratory of Cognitive Neuroscience and Learning, Beijing Normal University（认知神经科学与学习国家重点实验室，北京师范大学）

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL、cs.AI、cs.MM

2508.08066 2025-08-21 cs.CV cs.AI cs.CL cs.LG 85%

ExpVG: Investigating the Design Space of Visual Grounding in Multimodal Large Language Model

Weitai Kang, Weiming Zhuang, Zhizhong Li, Yan Yan, Lingjuan Lyu

机构 * University of Illinois Chicago（伊利诺伊大学香槟分校）； Sony AI（索尼人工智能）

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 8 pages for the main paper

2508.10371 2025-08-15 cs.RO 85%

Few-shot Vision-based Human Activity Recognition with MLLM-based Visual Reinforcement Learning

Wenqi Zheng, Yutaka Arakawa

机构 * Graduate School（研究生院）； Faculty of Information Science（信息科学系）； Electrical Engineering（电气工程）； Kyushu University（九州大学）； JAPAN（日本）

专题命中多模态训练与对齐：MLLM(title,abstract)；multimodal(abstract)；multi-modal(abstract)

2501.03012 2025-08-14 cs.AI cs.CL cs.CV 85%

Analyzing Finetuning Representation Shift for Multimodal LLMs Steering

Pegah Khayatan, Mustafa Shukor, Jayneel Parekh, Arnaud Dapogny, Matthieu Cord

机构 * ISIR, Sorbonne Université（ISIR，索邦大学）

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ICCV 2025. The first three authors contributed equally. Project page and code: https://pegah- kh.github.io/projects/lmm-finetuning-analysis-and-steering/

2407.14506 2025-07-21 cs.CV cs.AI cs.CL 85%

On Pre-training of Multimodal Language Models Customized for Chart Understanding

Wan-Cyuan Fan, Yen-Chun Chen, Mengchen Liu, Lu Yuan, Leonid Sigal

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NeurIPS 2024 Workshop on Adaptive Foundation Models

2507.07572 2025-07-11 cs.CL cs.AI cs.CV 85%

Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation

Yupu Liang, Yaping Zhang, Zhiyang Zhang, Yang Zhao, Lu Xiang, Chengqing Zong, Yu Zhou

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by ACL 2025 Main

2502.06355 2025-07-10 cs.DC cs.LG 85%

Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach

Timo Fudala, Vasileios Tsouvalas, Nirvana Meratnia

机构 * Eindhoven University of Technology（埃因霍温理工大学）

专题命中多模态训练与对齐：multimodal(title,abstract)；image-text(abstract)；audio-visual(abstract)

Comments 10 pages, 5 figures, accepted to FedGenAI-IJCAI 2025

2507.02985 2025-07-08 cs.CV cs.AI cs.CL 85%

Gated Recursive Fusion: A Stateful Approach to Scalable Multimodal Transformers

Yusuf Shihata

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 13 pages, 2 figures

2409.13609 2025-06-23 cs.CV cs.AI cs.CL 85%

MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension

Ting Liu, Zunnan Xu, Yue Hu, Liangtao Shi, Zhiqiang Wang, Quanjun Yin

机构 * College of Systems Engineering, National University of Defense Technology（系统工程学院，国防科技大学）； Tsinghua University（清华大学）； Hefei University of Technology（合肥工业大学）； iFLYTEK Research（iFLYTEK研究院）

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments EMNLP 2024 main

2408.05211 2025-06-02 cs.CV cs.AI cs.CL 85%

VITA: Towards Open-Source Interactive Omni Multimodal LLM

Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Yuhang Dai, Meng Zhao, Yi-Fan Zhang, Shaoqi Dong, Yangze Li, Xiong Wang, Haoyu Cao, Di Yin, Long Ma, Xiawu Zheng, Rongrong Ji, Yunsheng Wu, Ran He, Caifeng Shan, Xing Sun

机构 * NJU（南京大学）； Tencent Youtu Lab（腾讯计算机系统有限公司视觉实验室）； XMU（厦门大学）； CASIA（中国科学院自动化研究所）

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project Page: https://vita-home.github.io

2505.17316 2025-05-26 cs.CV cs.AI cs.CL cs.LG 85%

Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models

Jiachen Jiang, Jinxin Zhou, Bo Peng, Xia Ning, Zhihui Zhu

机构 * Department of Computer Science and Engineering, The Ohio State University（计算机科学与工程系，俄亥俄州立大学）； Translational Data Analytics Institute, The Ohio State University（转化数据分析研究所，俄亥俄州立大学）； Department of Biomedical Informatics, The Ohio State University（生物医学信息学系，俄亥俄州立大学）

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

2402.03607 2025-05-19 cs.AI cs.CL cs.CV cs.CY cs.HC 85%

Enhancing Cross-Modal Contextual Congruence for Crowdfunding Success using Knowledge-infused Learning

Trilok Padhi, Ugur Kursuncu, Yaman Kumar, Valerie L. Shalin, Lane Peterson Fronczek

机构 * Georgia State University（佐治亚州立大学）； Adobe MDSR ； Wright State University（怀特州立大学）； California Polytechnic State University（加州州立大学帕克校区）

专题命中多模态训练与对齐：cross-modal(title)；multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at IEEE International Conference on Big Data 2024 (IEEE BigData 2024)

Journal ref IEEE International Conference on Big Data 2024 (IEEE BigData 2024)

2505.10557 2025-05-16 cs.CV cs.AI cs.CL 85%

MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoning

Ke Wang, Junting Pan, Linda Wei, Aojun Zhou, Weikang Shi, Zimu Lu, Han Xiao, Yunqiao Yang, Houxing Ren, Mingjie Zhan, Hongsheng Li

机构 * Multimedia Laboratory (MMLab)（多媒体实验室）； The Chinese University of Hong Kong（香港中文大学）； CPII under InnoHK（创新香港科技署CPII）

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to ACL 2025 Findings

2504.17892 2025-04-28 cs.CV cs.AI cs.CL 85%

Token Sequence Compression for Efficient Multimodal Computing

Yasmine Omri, Parth Shroff, Thierry Tambe

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

2504.10738 2025-04-16 cs.CV cs.AI cs.CL cs.LG cs.RO 85%

CleanMAP: Distilling Multimodal LLMs for Confidence-Driven Crowdsourced HD Map Updates

Ankit Kumar Shaw, Kun Jiang, Tuopu Wen, Chandan Kumar Sah, Yining Shi, Mengmeng Yang, Diange Yang, Xiaoli Lian

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Kun Jiang, Mengmeng Yang and Diange Yang are Corresponding Author. The main paper and supplementary material are both included here, total 23 pages (main paper is 10 pages and supplementary material is 13 pages), total 17 figures (6 figures in main paper and 11 figures in supplementary material), this paper is Accepted to CVPR WDFM-AD Workshop 2025, The code will be available at https://Ankit-Zefan.github.io/CleanMap/

2504.10000 2025-04-15 cs.CR cs.AI cs.CL cs.CV cs.LG 85%

Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?

Yanbo Wang, Jiyang Guan, Jian Liang, Ran He

专题命中多模态训练与对齐：multi-modal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to CVPR 2025, codes in process

2404.16375 2025-01-22 cs.CV cs.AI cs.CL 85%

List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs

An Yan, Zhengyuan Yang, Junda Wu, Wanrong Zhu, Jianwei Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Julian McAuley, Jianfeng Gao, Lijuan Wang

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments published at COLM-2024

2409.10197 2024-12-30 cs.CV cs.CL cs.MM 85%

Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models

Weihao Ye, Qiong Wu, Wenhao Lin, Yiyi Zhou

专题命中多模态训练与对齐：multi-modal(title)；multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.MM

2406.08074 2024-12-03 cs.LG cs.AI cs.CL cs.CV 85%

A Concept-Based Explainability Framework for Large Multimodal Models

Jayneel Parekh, Pegah Khayatan, Mustafa Shukor, Alasdair Newson, Matthieu Cord

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NeurIPS 2024

2408.09438 2024-08-20 cs.MM cs.AI cs.CV cs.SD 85%

Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognition

Qifei Li, Yingming Gao, Yuhua Wen, Cong Wang, Ya Li

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments The paper has been accepted by INTERSPEECH 2024

2407.09029 2024-07-15 cs.MM cs.CV cs.SD eess.AS 85%

Enhancing Emotion Recognition in Incomplete Data: A Novel Cross-Modal Alignment, Reconstruction, and Refinement Framework

Haoqin Sun, Shiwan Zhao, Shaokai Li, Xiangyu Kong, Xuechen Wang, Aobo Kong, Jiaming Zhou, Yong Chen, Wenjia Zeng, Yong Qin

专题命中多模态训练与对齐：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

2405.20797 2024-06-18 cs.CV cs.AI cs.CL cs.LG 85%

Ovis: Structural Embedding Alignment for Multimodal Large Language Model

Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Han-Jia Ye

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

2406.09412 2024-06-14 cs.CV cs.AI cs.LG cs.MM 85%

Explore the Limits of Omni-modal Pretraining at Scale

Yiyuan Zhang, Handong Li, Jing Liu, Xiangyu Yue

专题命中多模态训练与对齐：omni-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Project Website: https://invictus717.github.io/MiCo/

2311.17647 2024-06-12 cs.CV cs.AI cs.CL 85%

Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?

Xiujun Li, Yujie Lu, Zhe Gan, Jianfeng Gao, William Yang Wang, Yejin Choi

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Github: https://github.com/VIM-Bench/VIM_TOOL, Model and Data: https://huggingface.co/VIM-Bench

2405.17459 2024-05-29 cs.LG cs.AI cs.CL cs.CV 85%

Integrating Medical Imaging and Clinical Reports Using Multimodal Deep Learning for Advanced Disease Analysis

Ziyan Yao, Fei Lin, Sheng Chai, Weijie He, Lu Dai, Xinghui Fei

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

2312.06742 2024-04-02 cs.CV cs.AI cs.CL cs.LG 85%

Honeybee: Locality-enhanced Projector for Multimodal LLM

Junbum Cha, Wooyoung Kang, Jonghwan Mun, Byungseok Roh

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2024 camera-ready

2403.09072 2024-03-15 cs.CV cs.AI cs.CL 85%

UniCode: Learning a Unified Codebook for Multimodal Large Language Models

Sipeng Zheng, Bohan Zhou, Yicheng Feng, Ye Wang, Zongqing Lu

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 14 pages, 2 figures, 11 tables

2312.07212 2023-12-13 cs.MM cs.AI cs.SD eess.AS 85%

More than Vanilla Fusion: a Simple, Decoupling-free, Attention Module for Multimodal Fusion Based on Signal Theory

Peiwen Sun, Yifan Zhang, Zishan Liu, Donghao Chen, Honggang Zhang

专题命中多模态训练与对齐：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.AI、cs.MM、eess.AS

2307.11845 2023-11-28 cs.CL cs.AI cs.CV q-fin.CP 85%

Multimodal Document Analytics for Banking Process Automation

Christopher Gerling, Stefan Lessmann

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments A Preprint

1. 多模态训练与对齐 6897 篇

LFTR: Learning-Free Token Reduction for Multimodal Large Language Models

Bridging the behavior-neural gap: A multimodal AI reveals the brain's geometry of emotion more accurately than human self-reports

ExpVG: Investigating the Design Space of Visual Grounding in Multimodal Large Language Model

Few-shot Vision-based Human Activity Recognition with MLLM-based Visual Reinforcement Learning

Analyzing Finetuning Representation Shift for Multimodal LLMs Steering

On Pre-training of Multimodal Language Models Customized for Chart Understanding

Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation

Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach

Gated Recursive Fusion: A Stateful Approach to Scalable Multimodal Transformers

MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension

VITA: Towards Open-Source Interactive Omni Multimodal LLM

Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models

Enhancing Cross-Modal Contextual Congruence for Crowdfunding Success using Knowledge-infused Learning

MathCoder-VL: Bridging Vision and Code for Enhanced Multimodal Mathematical Reasoning

Token Sequence Compression for Efficient Multimodal Computing

CleanMAP: Distilling Multimodal LLMs for Confidence-Driven Crowdsourced HD Map Updates

Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?

List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs

Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models

A Concept-Based Explainability Framework for Large Multimodal Models

Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognition

Enhancing Emotion Recognition in Incomplete Data: A Novel Cross-Modal Alignment, Reconstruction, and Refinement Framework

Ovis: Structural Embedding Alignment for Multimodal Large Language Model

Explore the Limits of Omni-modal Pretraining at Scale

Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?

Integrating Medical Imaging and Clinical Reports Using Multimodal Deep Learning for Advanced Disease Analysis

Honeybee: Locality-enhanced Projector for Multimodal LLM

UniCode: Learning a Unified Codebook for Multimodal Large Language Models

More than Vanilla Fusion: a Simple, Decoupling-free, Attention Module for Multimodal Fusion Based on Signal Theory

Multimodal Document Analytics for Banking Process Automation