多模态大模型 - arXivDaily 专题

2204.01678 2022-04-05 cs.CV cs.LG 83%

MultiMAE: Multi-modal Multi-task Masked Autoencoders

Roman Bachmann, David Mizrahi, Andrei Atanov, Amir Zamir

专题命中多模态评测：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Project page at https://multimae.epfl.ch

2105.07122 2022-03-18 cs.CL 83%

Premise-based Multimodal Reasoning: Conditional Inference on Joint Textual and Visual Clues

Qingxiu Dong, Ziwei Qin, Heming Xia, Tian Feng, Shoujie Tong, Haoran Meng, Lin Xu, Weidong Zhan, Sujian Li, Zhongyu Wei, Tianyu Liu, Zuifang Sui

专题命中多模态评测：multimodal(title)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CL

Comments ACL 2022 Main conference (Long Paper)

2105.14517 2022-01-12 cs.AI 83%

GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning

Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P. Xing, Liang Lin

专题命中多模态评测：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI

Comments Accepted to Findings of ACL 2021

2201.02242 2022-01-10 eess.IV cs.CV 83%

A Keypoint Detection and Description Network Based on the Vessel Structure for Multi-Modal Retinal Image Registration

Aline Sindel, Bettina Hohberger, Sebastian Fassihi Dehcordi, Christian Mardin, Robert Lämmer, Andreas Maier, Vincent Christlein

专题命中多模态评测：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments 6 pages, 4 figures, 1 table, accepted to BVM 2022

2112.04873 2021-12-10 cs.CL 83%

Nice perfume. How long did you marinate in it? Multimodal Sarcasm Explanation

Poorav Desai, Tanmoy Chakraborty, Md Shad Akhtar

专题命中多模态评测：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL

Comments Accepted for publication in AAAI-2022

2111.04264 2021-11-12 cs.CV 83%

Cross-Modal Object Tracking: Modality-Aware Representations and A Unified Benchmark

Chenglong Li, Tianhao Zhu, Lei Liu, Xiaonan Si, Zilin Fan, Sulan Zhai

专题命中多模态评测：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments In Submission

2104.05971 2021-04-14 cs.CV 83%

Learning Multi-modal Information for Robust Light Field Depth Estimation

Yongri Piao, Xinxin Ji, Miao Zhang, Yukun Zhang

专题命中多模态评测：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2103.02636 2021-03-05 cs.CL cs.LG 83%

A Novel Context-Aware Multimodal Framework for Persian Sentiment Analysis

Kia Dashtipour, Mandar Gogate, Erik Cambria, Amir Hussain

专题命中多模态评测：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL

Comments Accepted in Neurocomputing

2010.12126 2020-10-29 cs.CV 83%

Beyond the Deep Metric Learning: Enhance the Cross-Modal Matching with Adversarial Discriminative Domain Regularization

Li Ren, Kai Li, LiQiang Wang, Kien Hua

专题命中多模态评测：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments 8 pages

1905.08789 2020-07-16 cs.CV 83%

Borrow from Anywhere: Pseudo Multi-modal Object Detection in Thermal Imagery

Chaitanya Devaguptapu, Ninad Akolekar, Manuj M Sharma, Vineeth N Balasubramanian

专题命中多模态评测：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments Accepted at Perception Beyond Visible Spectrum Workshop, CVPR 2019

1911.07731 2020-05-29 eess.IV cs.CV 83%

Multi-modal Deep Guided Filtering for Comprehensible Medical Image Processing

Bernhard Stimpel, Christopher Syben, Franziska Schirrmacher, Philipp Hoelter, Arnd Dörfler, Andreas Maier

专题命中多模态评测：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Journal ref IEEE Transactions on Medical Imaging, vol. 39, no. 5, pp. 1703-1711, May 2020

2004.12238 2020-04-28 cs.CL 83%

MCQA: Multimodal Co-attention Based Network for Question Answering

Abhishek Kumar, Trisha Mittal, Dinesh Manocha

专题命中多模态评测：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL

1911.12676 2020-04-01 cs.CV 83%

xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation

Maximilian Jaritz, Tuan-Hung Vu, Raoul de Charette, Émilie Wirbel, Patrick Pérez

专题命中多模态评测：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted at CVPR 2020. For a demo video, see http://tiny.cc/xmuda

2002.02649 2020-02-10 cs.CL 83%

Multimodal Matching Transformer for Live Commenting

Chaoqun Duan, Lei Cui, Shuming Ma, Furu Wei, Conghui Zhu, Tiejun Zhao

专题命中多模态评测：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL

1907.08948 2019-07-23 cs.CL 83%

Hindi Visual Genome: A Dataset for Multimodal English-to-Hindi Machine Translation

Shantipriya Parida, Ondřej Bojar, Satya Ranjan Dash

专题命中多模态评测：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CL

Comments 6 pages, 3 figures

1906.06322 2019-06-17 cs.CV cs.LG cs.RO 83%

Connecting Touch and Vision via Cross-Modal Prediction

Yunzhu Li, Jun-Yan Zhu, Russ Tedrake, Antonio Torralba

专题命中多模态评测：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted to CVPR 2019. Project Page: http://visgel.csail.mit.edu/

1901.02858 2019-01-10 cs.CV 83%

Adaptive Feature Processing for Robust Human Activity Recognition on a Novel Multi-Modal Dataset

Mirco Moencks, Varuna De Silva, Jamie Roche, Ahmet Kondoz

专题命中多模态评测：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments Working Draft

1602.08225 2016-02-29 cs.HC cs.CV cs.LG 83%

Multimodal Emotion Recognition Using Multimodal Deep Learning

Wei Liu, Wei-Long Zheng, Bao-Liang Lu

专题命中多模态评测：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2606.05008 2026-06-04 cs.CV cs.AI cs.CL 83%

M$^3$Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks

M$^3$Eval: 通过认知基础视频任务的多模态记忆评估

Jie Huang, Ruixun Liu, Sirui Sun, Xinyi Yang, Yin Li, Yixin Zhu, Yiwu Zhong

机构 * School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）； State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）； Yuanpei College, Peking University（北京大学元培学院）； Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）； School of Psychological and Cognitive Sciences, Peking University（北京大学心理学与认知科学学院）； University of Wisconsin-Madison（威斯康星大学麦迪逊分校）

专题命中多模态评测：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结提出首个多模态模型记忆评估框架M$^3$Eval，通过认知心理学设计的视频任务系统评估模型在记忆保持、忠实性和鲁棒性上的表现，发现模型在并行视频流处理、干扰模式、时空记忆和符号记忆方面的显著缺陷。

Comments We present an evaluation designed for multi-modal memory in multi-modal models

2510.07632 2026-04-27 cs.AI cs.CL cs.CV cs.LG 83%

Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models

测试时匹配：在多模态模型中解锁组合推理

Yinglun Zhu, Jiancheng Zhang, Fuzhi Tang

机构 * University of California, Riverside（加州大学河滨分校）

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结本文提出测试时匹配算法，通过改进评估指标提升多模态模型的组合推理能力，使SigLIP-B16和GPT-4.1在Winoground等基准上取得新突破。

Comments To appear at ICLR 2026; extended results to generative multimodal models

2505.14707 2025-05-22 cs.MM cs.AI cs.CV 83%

CrypticBio: A Large Multimodal Dataset for Visually Confusing Biodiversity

Georgiana Manolache, Gerard Schouten, Joaquin Vanschoren

机构 * Fontys University of Applied Sciences（Fontys应用科学大学）； Technical University of Eindhoven（埃因霍温技术大学）

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments We present CrypticBio, the largest publicly available multimodal dataset of visually confusing species, specifically curated to support the development of AI models for biodiversity identification using images, language and spatiotemporal data

2407.04973 2024-07-09 cs.AI cs.CL cs.CV cs.LG 83%

LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts

Yijia Xiao, Edward Sun, Tianyu Liu, Wei Wang

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments LogicVista benchmarks the logical reasoning of multimodal large language models in visual tasks

2404.12390 2024-07-04 cs.CV cs.AI cs.CL 83%

BLINK: Multimodal Large Language Models Can See but Not Perceive

Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, Ranjay Krishna

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Multimodal Benchmark, Project Url: https://zeyofu.github.io/blink/, ECCV 2024

2607.27378 2026-07-31 cs.CV cs.MM 新提交 82%

PanDent: Toward Comprehensive Tooth-Level Structure-Language Consistency in Dental Radiology

PanDent：面向牙科放射学中全面的牙级结构-语言一致性

Xiaohan Li, Xinyu Liu, Chang Liu, Sum Wing Au Yeung, Jun Liu, Yixuan Yuan, Hui Chen

机构 * Faculty of Dentistry, The University of Hong Kong（香港大学牙医学院）； Imperial College London（帝国理工学院）； University of Science and Technology of China（中国科学技术大学）； Department of Data and Systems Engineering, The University of Hong Kong（香港大学数据与系统工程系）； Department of Electronic Engineering, The Chinese University of Hong Kong（香港中文大学电子工程系）

专题命中多模态评测：MLLM(summary_cn,abstract_cn)；multimodal(abstract)；分类 cs.CV、cs.MM

AI总结本研究推出PanDent牙科OPG基准，经实验发现现有MLLM生成的牙科报告流畅但临床一致性差，在PanDent上微调可提升其结构-语言一致性，该基准可用于评估MLLM的牙级临床推理能力。

2607.09654 2026-07-13 cs.CV cs.AI 新提交 82%

Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models

十年视觉语言人工智能模型中准确性和视觉认知错误的演变

Shravan Murlidaran, Miguel P. Eckstein

机构 * Psychological & Brain Sciences, University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校心理与脑科学系）； Department of Computer Science, University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校计算机科学系）； Department of Electrical and Computer Engineering, University of California, Santa Barbara（加利福尼亚大学圣巴巴拉分校电气与计算机工程系）

专题命中多模态评测：MLLM(summary_cn,abstract_cn)；multimodal(abstract)；分类 cs.CV、cs.AI

AI总结研究十年间视觉语言模型进展，引入CSB数据集，评估模型在其上及MS-COCO样本中的准确性与视觉认知错误类型，发现MLLM消除简单与复杂场景描述准确性差距，几乎消除多数错误类型，为模型发展提供全面评估。

2605.28215 2026-07-13 cs.AI cs.CL cs.LG cs.LO cs.MA 版本更新 82%

Explaining is Harder Than Predicting Alone: Evaluating Concept-based Explanations of MLLMs as ICL Visual Classifiers

解释比单独预测更难：评估基于概念的MLLM解释作为ICL视觉分类器

Carmen Quiles-Ramírez, Leticia L. Rodríguez, Nicolás Martorell, Natalia Díaz-Rodríguez

专题命中多模态评测：MLLM(title_cn,abstract_cn)；multimodal(abstract)；分类 cs.CL、cs.AI

AI总结本文通过五种形式化程度递增的条件，系统评估多模态大语言模型在少样本上下文学习中的基于概念的可解释性，发现解释比预测更难，且强制生成形式化解释会降低预测准确性。

Comments Accepted to the CompLearn Workshop at ICML 2026

2602.07045 2026-05-15 cs.CV cs.AI 82%

VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensing

VLRS-Bench: 一种面向遥感的视觉-语言推理基准

Zhiming Luo, Di Wang, Haonan Guo, Jing Zhang, Bo Du

机构 * School of Computer Science, Wuhan University（武汉大学计算机学院）

专题命中多模态评测：MLLM(summary_cn,abstract_cn)；multimodal(abstract)；分类 cs.CV、cs.AI

AI总结本文提出VLRS-Bench，首个专注于复杂遥感推理的基准，包含2000个问题-答案对，涵盖14项任务和八个时间阶段，揭示现有MLLM在遥感任务中的瓶颈。

2605.08800 2026-05-12 cs.CV cs.AI 82%

PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Models

PPU-Bench: 用于视觉语言模型个性化部分遗忘的现实世界基准

Jiahui Guang, Zexun Zhan, Zhenlin Xu, Cuiyun Gao, Haiyan Wang, Jing Li, Zhaoquan Gu, Yanchun Zhang

机构 * Harbin Institute of Technology（哈尔滨工业大学）； Pengcheng Laboratory（鹏城实验室）； The Hong Kong Polytechnic University（香港理工大学）； Sichuan University（四川大学）； Zhejiang Normal University（浙江师范大学）

专题命中多模态评测：MLLM(abstract,abstract_cn)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

AI总结本文提出PPU-Bench，一个无需微调的现实世界基准，用于评估视觉语言模型中个性化部分遗忘的效果，通过24K多模态和单模态样本测试遗忘与保留的平衡及跨模态一致性。

2312.12436 2023-12-21 cs.CV cs.AI cs.CL cs.MM 82%

A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise

Chaoyou Fu, Renrui Zhang, Zihan Wang, Yubo Huang, Zhengye Zhang, Longtian Qiu, Gaoxiang Ye, Yunhang Shen, Mengdan Zhang, Peixian Chen, Sirui Zhao, Shaohui Lin, Deqiang Jiang, Di Yin, Peng Gao, Ke Li, Hongsheng Li, Xing Sun

专题命中多模态评测：multimodal(abstract,comments)；multi-modal(abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Total 120 pages. See our project at https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models

2506.16633 2026-08-24 cs.CL cs.AI cs.MM 版本更新 82%

GeoExplain: Multimodal Reasoning based on Hierarchy of Visual Information in Street View

GeoExplain：基于街景图像视觉信息层次的多模态推理

Fenghua Cheng, Jinxiang Wang, Sen Wang, Zi Huang, Xue Li

机构 * The University of Queensland（昆士兰大学）

专题命中多模态评测：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

AI总结针对现有多模态推理任务缺乏对不同粒度视觉线索推理的研究缺口，提出GeoExplain数据集与SightSense方法，实现街景图像的地理定位预测与可解释性说明，且方法在该数据集上表现优异。

Comments Updated version

1. 多模态评测 9150 篇

MultiMAE: Multi-modal Multi-task Masked Autoencoders

Premise-based Multimodal Reasoning: Conditional Inference on Joint Textual and Visual Clues

GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning

A Keypoint Detection and Description Network Based on the Vessel Structure for Multi-Modal Retinal Image Registration

Nice perfume. How long did you marinate in it? Multimodal Sarcasm Explanation

Cross-Modal Object Tracking: Modality-Aware Representations and A Unified Benchmark

Learning Multi-modal Information for Robust Light Field Depth Estimation

A Novel Context-Aware Multimodal Framework for Persian Sentiment Analysis

Beyond the Deep Metric Learning: Enhance the Cross-Modal Matching with Adversarial Discriminative Domain Regularization

Borrow from Anywhere: Pseudo Multi-modal Object Detection in Thermal Imagery

Multi-modal Deep Guided Filtering for Comprehensible Medical Image Processing

MCQA: Multimodal Co-attention Based Network for Question Answering

xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation

Multimodal Matching Transformer for Live Commenting

Hindi Visual Genome: A Dataset for Multimodal English-to-Hindi Machine Translation

Connecting Touch and Vision via Cross-Modal Prediction

Adaptive Feature Processing for Robust Human Activity Recognition on a Novel Multi-Modal Dataset

Multimodal Emotion Recognition Using Multimodal Deep Learning

M$^3$Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks

Test-Time Matching: Unlocking Compositional Reasoning in Multimodal Models

CrypticBio: A Large Multimodal Dataset for Visually Confusing Biodiversity

LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts

BLINK: Multimodal Large Language Models Can See but Not Perceive

PanDent: Toward Comprehensive Tooth-Level Structure-Language Consistency in Dental Radiology

Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models

Explaining is Harder Than Predicting Alone: Evaluating Concept-based Explanations of MLLMs as ICL Visual Classifiers

VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensing

PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Models

A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise

GeoExplain: Multimodal Reasoning based on Hierarchy of Visual Information in Street View