多模态大模型 - arXivDaily 专题

2505.05422 2025-08-18 cs.CV cs.AI cs.CL 82%

TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation

Haokun Lin, Teng Wang, Yixiao Ge, Yuying Ge, Zhichao Lu, Ying Wei, Qingfu Zhang, Zhenan Sun, Ying Shan

机构 * ARC Lab, Tencent PCG（腾讯PCG广告实验室）； City University of Hong Kong（香港城市大学）； Zhejiang University（浙江大学）； NLPR & MAIS, Institute of Automation, CAS, Beijing（自动化研究所）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Technical Report

2507.04377 2025-08-12 cs.CV cs.CL cs.MM 82%

Multi-Modal Semantic Parsing for the Interpretation of Tombstone Inscriptions

Xiao Zhang, Johan Bos

机构 * University of Groningen（格罗宁根大学）

专题命中图文多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments ACMMM 2025

2508.05684 2025-08-11 cs.CR cs.LG 82%

MM-FusionNet: Context-Aware Dynamic Fusion for Multi-modal Fake News Detection with Large Vision-Language Models

Junhao He, Tianyu Liu, Jingyuan Zhao, Benjamin Turner

机构 * Huaiyin Institute of Technology（淮阴职业技术学院）； Universidad Autónoma de Asunción（阿斯unción自治大学）

专题命中图文多模态：multi-modal(title,abstract)；cross-modal(abstract)

2410.08642 2025-08-05 cs.SI cs.CL cs.CV cs.MM 82%

More than Memes: A Multimodal Topic Modeling Approach to Conspiracy Theories on Telegram

Elisabeth Steffen

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments 12 pages, 10 figures

Journal ref Proceedings of the Nineteenth International AAAI Conference on Web and Social Media, Vol. 19 (2025)

2507.07151 2025-07-11 cs.CV cs.AI cs.CL 82%

Robust Multimodal Large Language Models Against Modality Conflict

Zongmeng Zhang, Wengang Zhou, Jie Zhao, Houqiang Li

机构 * School of Artificial Intelligence（人工智能学院）； Data Science, University of Science（数据科学，科学大学）； Department of Electronic Engineering（电子工程系）； Information Science, University of Science（信息科学，科学大学）； Huawei Technologies Co., Ltd.（华为技术有限公司）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ICML 2025

2503.04065 2025-06-27 cs.CV cs.AI cs.CL 82%

PP-DocBee: Improving Multimodal Document Understanding Through a Bag of Tricks

Feng Ni, Kui Huang, Yao Lu, Wenyu Lv, Guanzhong Wang, Zeyu Chen, Yi Liu

机构 * PaddlePaddle Team, Baidu Inc.（百度公司）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2504.17353 2025-06-13 cs.CL cs.CV cs.MM 82%

M-MRE: Extending the Mutual Reinforcement Effect to Multimodal Information Extraction

Chengguang Gan, Zhixi Cai, Yanbin Wei, Yunhao Liang, Shiwen Ni, Tatsunori Mori

机构 * Yokohama National University（Yokohama国立大学）； Monash University（墨尔本大学）； Southern University of Science and Technology（南方科技大学）； Hong Kong University of Science and Technology（香港科学大学）； University of Chinese Academy of Sciences（中国科学院大学）； Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

2506.07936 2025-06-10 cs.CV cs.AI cs.CL cs.LG 82%

Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models

Chengyue Huang, Yuchen Zhu, Sichen Zhu, Jingyun Xiao, Moises Andrade, Shivang Chopra, Zsolt Kira

机构 * Georgia Institute of Technology（佐治亚理工学院）

专题命中图文多模态：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

2412.00142 2025-06-10 cs.CV cs.AI cs.CL 82%

Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features

Chancharik Mitra, Brandon Huang, Tianning Chai, Zhiqiu Lin, Assaf Arbelle, Rogerio Feris, Leonid Karlinsky, Trevor Darrell, Deva Ramanan, Roei Herzig

机构 * Carnegie Mellon University（卡内基梅隆大学）； University of California, Berkeley（加州大学伯克利分校）； IBM Research（IBM研究院）； MIT-IBM Watson AI Lab（麻省理工-IBM沃森人工智能实验室）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2412.06141 2025-06-05 cs.CV cs.AI cs.CL cs.LG 82%

MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization

Kangyu Zhu, Peng Xia, Yun Li, Hongtu Zhu, Sheng Wang, Huaxiu Yao

机构 * UNC Chapel-Hill（UNC夏洛特-希尔分校）； Brown University（布朗大学）； University of Washington（华盛顿大学）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ICML 2025

2503.14604 2025-06-02 cs.CV cs.AI cs.CL 82%

Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives

Sara Sarto, Marcella Cornia, Rita Cucchiara

机构 * University of Modena and Reggio Emilia（摩德纳和雷吉奥艾米利亚大学）； IIT-CNR（意大利国家研究委员会IIT）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments IJCAI 2025. Repo GitHub: https://github.com/aimagelab/awesome-captioning-evaluation

2505.22943 2025-05-30 cs.CL cs.AI cs.CV cs.LG cs.SD 82%

Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of Pre-trained Multimodal Representation via Text Updates

Jaewoo Ahn, Heeseung Yun, Dayoon Ko, Gunhee Kim

机构 * Seoul National University（首尔国立大学）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL 2025 Main. Code is released at https://vision.snu.ac.kr/projects/mac

2505.16446 2025-05-23 cs.LG 82%

Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Models

Zhaoxin Wang, Handing Wang, Cong Tian, Yaochu Jin

机构 * Xidian University（西安电子科技大学）； Westlake University（西湖大学）

专题命中图文多模态：cross-modal(title,abstract)；multimodal(abstract)

2505.05040 2025-05-09 cs.CL cs.AI cs.CV 82%

Image-Text Relation Prediction for Multilingual Tweets

Matīss Rikters, Edison Marrese-Taylor

机构 * Artificial Intelligence Research Center（人工智能研究中心）； National Institute of Advanced Industrial Science and Technology（国家先进工业科学与技术研究院）

专题命中图文多模态：image-text(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Journal ref Published in Proceedings of the 1st Workshop on Nordic-Baltic Responsible Evaluation and Alignment of Language, NoDaLiDa - Baltic HLT 2025

2505.04911 2025-05-09 cs.CV cs.AI cs.CL 82%

SpatialPrompting: Keyframe-driven Zero-Shot Spatial Reasoning with Off-the-Shelf Multimodal Large Language Models

Shun Taguchi, Hideki Deguchi, Takumi Hamazaki, Hiroyuki Sakai

机构 * Toyota Central R&D Labs., Inc.（丰田中央研究实验室）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 18 pages, 11 figures

2505.00150 2025-05-02 cs.CV cs.AI cs.CL 82%

Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models

Minh-Hao Van, Xintao Wu

机构 * Department of EECS University of Arkanasas Fayetteville（电子工程与科学系阿肯色大学法洛维尔分校）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2412.00876 2025-03-24 cs.CV cs.AI cs.CL cs.LG 82%

Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification

Wenxuan Huang, Zijie Zhai, Yunhang Shen, Shaosheng Cao, Fei Zhao, Xiangfeng Xu, Zheyu Ye, Yao Hu, Shaohui Lin

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to ICLR 2025. Code is available at https://github.com/Osilly/dynamic_llava

2403.09974 2025-03-24 cs.CV cs.AI cs.CL cs.LG 82%

GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discovery

Enguang Wang, Zhimao Peng, Zhengyuan Xie, Fei Yang, Xialei Liu, Ming-Ming Cheng

专题命中图文多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2025

2503.10582 2025-03-18 cs.CV cs.AI cs.CL 82%

VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search

Yiming Jia, Jiachen Li, Xiang Yue, Bo Li, Ping Nie, Kai Zou, Wenhu Chen

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Technical Report

2503.10857 2025-03-17 cs.GR cs.AI cs.CL cs.CV 82%

Towards Understanding Graphical Perception in Large Multimodal Models

Kai Zhang, Jianwei Yang, Jeevana Priya Inala, Chandan Singh, Jianfeng Gao, Yu Su, Chenglong Wang

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Work in Progress

2502.00094 2025-02-06 cs.CV cs.AI cs.CL cs.HC cs.LG 82%

AIN: The Arabic INclusive Large Multimodal Model

Ahmed Heakl, Sara Ghaboura, Omkar Thawkar, Fahad Shahbaz Khan, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 20 pages, 16 figures, ACL

2410.10879 2024-12-11 cs.LG cs.AI cs.CL cs.CV 82%

Enhancing Vision-Language Model Pre-training with Image-text Pair Pruning Based on Word Frequency

Mingliang Liang, Martha Larson

专题命中图文多模态：image-text(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2411.11758 2024-11-19 cs.CV cs.AI cs.CL 82%

The Power of Many: Multi-Agent Multimodal Models for Cultural Image Captioning

Longju Bai, Angana Borah, Oana Ignat, Rada Mihalcea

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2405.07001 2024-11-07 cs.CL cs.AI cs.CV 82%

ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering

Yifan Wu, Lutao Yan, Leixian Shen, Yunhai Wang, Nan Tang, Yuyu Luo

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments EMNLP 2024 Conference Paper

2410.15509 2024-10-22 cs.LG cs.AI cs.CL cs.CV 82%

Exploring Curriculum Learning for Vision-Language Tasks: A Study on Small-Scale Multimodal Training

Rohan Saha, Abrar Fahim, Alona Fyshe, Alex Murphy

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CoNLL BabyLM Challenge 2024 camera ready

2410.05210 2024-10-08 cs.CV cs.AI cs.CL 82%

Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality

Youngtaek Oh, Jae Won Cho, Dong-Jin Kim, In So Kweon, Junmo Kim

专题命中图文多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments EMNLP 2024 (Long, Main). Project page: https://ytaek-oh.github.io/fsc-clip

2409.13592 2024-09-23 cs.CV cs.AI cs.CL 82%

YesBut: A High-Quality Annotated Multimodal Dataset for evaluating Satire Comprehension capability of Vision-Language Models

Abhilash Nandy, Yash Agarwal, Ashish Patwa, Millon Madhur Das, Aman Bansal, Ankit Raj, Pawan Goyal, Niloy Ganguly

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments EMNLP 2024 Main (Long), 18 pages, 14 figures, 12 tables

2311.06607 2024-08-27 cs.CV cs.AI cs.CL 82%

Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models

Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, Xiang Bai

专题命中图文多模态：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2024 Highlight

2408.12808 2024-08-26 cs.CV cs.AI cs.CL cs.LG 82%

VALE: A Multimodal Visual and Language Explanation Framework for Image Classifiers using eXplainable AI and Language Models

Purushothaman Natarajan, Athira Nambiar

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 15 pages, 10 tables, 3 figures

2408.08709 2024-08-19 cs.IR 82%

Multimodal Relational Triple Extraction with Query-based Entity Object Transformer

Lei Hei, Ning An, Tingjing Liao, Qi Ma, Jiaqi Wang, Feiliang Ren

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)

Comments 15 pages, 7 figures, preprint

1. 图文多模态 4657 篇

TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation

Multi-Modal Semantic Parsing for the Interpretation of Tombstone Inscriptions

MM-FusionNet: Context-Aware Dynamic Fusion for Multi-modal Fake News Detection with Large Vision-Language Models

More than Memes: A Multimodal Topic Modeling Approach to Conspiracy Theories on Telegram

Robust Multimodal Large Language Models Against Modality Conflict

PP-DocBee: Improving Multimodal Document Understanding Through a Bag of Tricks

M-MRE: Extending the Mutual Reinforcement Effect to Multimodal Information Extraction

Mimicking or Reasoning: Rethinking Multi-Modal In-Context Learning in Vision-Language Models

Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features

MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization

Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives

Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of Pre-trained Multimodal Representation via Text Updates

Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Models

Image-Text Relation Prediction for Multilingual Tweets

SpatialPrompting: Keyframe-driven Zero-Shot Spatial Reasoning with Off-the-Shelf Multimodal Large Language Models

Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models

Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification

GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discovery

VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search

Towards Understanding Graphical Perception in Large Multimodal Models

AIN: The Arabic INclusive Large Multimodal Model

Enhancing Vision-Language Model Pre-training with Image-text Pair Pruning Based on Word Frequency

The Power of Many: Multi-Agent Multimodal Models for Cultural Image Captioning

ChartInsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering

Exploring Curriculum Learning for Vision-Language Tasks: A Study on Small-Scale Multimodal Training

Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality

YesBut: A High-Quality Annotated Multimodal Dataset for evaluating Satire Comprehension capability of Vision-Language Models

Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models

VALE: A Multimodal Visual and Language Explanation Framework for Image Classifiers using eXplainable AI and Language Models

Multimodal Relational Triple Extraction with Query-based Entity Object Transformer