多模态大模型

A Survey of Multimodal Composite Editing and Retrieval

Suyan Li, Fuxiang Huang, Lei Zhang

专题命中 跨模态检索 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 20 pages, 3 figures, and 11 tables

PC$^2$: Pseudo-Classification Based Pseudo-Captioning for Noisy Correspondence Learning in Cross-Modal Retrieval

Yue Duan, Zhangxuan Gu, Zhenzhe Ying, Lei Qi, Changhua Meng, Yinghuan Shi

专题命中 跨模态检索 ：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by ACM MM 2024

SciMMIR: Benchmarking Scientific Multi-modal Information Retrieval

Siwei Wu, Yizhi Li, Kang Zhu, Ge Zhang, Yiming Liang, Kaijing Ma, Chenghao Xiao, Haoran Zhang, Bohao Yang, Wenhu Chen, Wenhao Huang, Noura Al Moubayed, Jie Fu, Chenghua Lin

专题命中 跨模态检索 ：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments camera-ready version for ACL 2024 Findings

Learning to Rematch Mismatched Pairs for Robust Cross-Modal Retrieval

Haochen Han, Qinghua Zheng, Guang Dai, Minnan Luo, Jingdong Wang

专题命中 跨模态检索 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments CVPR 2024

MM-Point: Multi-View Information-Enhanced Multi-Modal Self-Supervised 3D Point Cloud Understanding

Hai-Tao Yu, Mofei Song

专题命中 跨模态检索 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by AAAI 2024

Journal ref AAAI 2024

InvGC: Robust Cross-Modal Retrieval by Inverse Graph Convolution

Xiangru Jian, Yimu Wang

专题命中 跨模态检索 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Findings of EMNLP 2023

An Empirical Study of Multimodal Model Merging

Yi-Lin Sung, Linjie Li, Kevin Lin, Zhe Gan, Mohit Bansal, Lijuan Wang

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments EMNLP 2023 Findings

MACO: A Modality Adversarial and Contrastive Framework for Modality-missing Multi-modal Knowledge Graph Completion

Yichi Zhang, Zhuo Chen, Wen Zhang

专题命中 跨模态检索 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI、cs.MM

Comments This is the ArXiv version of our paper accepted by NLPCC 2023. The code will be released soon

Cross-Modal Attribute Insertions for Assessing the Robustness of Vision-and-Language Learning

Shivaen Ramshetty, Gaurav Verma, Srijan Kumar

专题命中 跨模态检索 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted full paper at ACL 2023; 15 pages, 7 figures

TOT: Topology-Aware Optimal Transport For Multimodal Hate Detection

Linhao Zhang, Li Jin, Xian Sun, Guangluan Xu, Zequn Zhang, Xiaoyu Li, Nayu Liu, Qing Liu, Shiyao Yan

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI、cs.MM

Comments accepted at AAAI23

MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text

Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, William W. Cohen

专题命中 跨模态检索 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to EMNLP 2022 main conference

A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval

Zhixiong Zeng, Wenji Mao

专题命中 跨模态检索 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.MM

Cross-modal Zero-shot Hashing by Label Attributes Embedding

Runmin Wang, Guoxian Yu, Lei Liu, Lizhen Cui, Carlotta Domeniconi, Xiangliang Zhang

专题命中 跨模态检索 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 7 pages, 2 figures

Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers

Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, Jianlong Fu

专题命中 跨模态检索 ：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.MM

Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation

Mohammad Mahdi Abootorabi, Amirhosein Zobeiri, Mahdi Dehghani, Mohammadali Mohammadkhani, Bardia Mohammadi, Omid Ghahroodi, Mahdieh Soleymani Baghshah, Ehsaneddin Asgari

机构 * Computer Engineering Department, Sharif University of Technology（沙斐大学计算机工程系） ； College of Interdisciplinary Science and Technology, University of Tehran（德黑兰大学跨学科科学与技术学院） ； Computer Engineering Department, K.N. Toosi University of Technology（技术学院计算机工程系） ； Qatar Computing Research Institute（卡塔尔计算研究院） 纠错

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI

Comments GitHub repository: https://github.com/llm-lab-org/Multimodal-RAG-Survey

Joint Wasserstein Autoencoders for Aligning Multimodal Embeddings

Shweta Mahajan, Teresa Botschen, Iryna Gurevych, Stefan Roth

专题命中 跨模态检索 ：multimodal(title)；cross-modal(abstract,comments)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Accepted at ICCV 2019 Workshop on Cross-Modal Learning in Real World

The Transparent Earth: A Multimodal Foundation Model for the Earth's Subsurface

Arnab Mazumder, Javier E. Santos, Noah Hobbs, Mohamed Mehana, Daniel O'Malley

机构 * Energy and Natural Resources Security Group（能源与自然资源安全组） ； Los Alamos National Laboratory（洛斯阿拉莫斯国家实验室） 纠错

专题命中 跨模态检索 ：multimodal(title)；multimodal foundation model(title)；分类 cs.AI

Comments Accepted at the Neurips 2025 AI4Science Workshop

Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs

Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI；multimodal foundation model(comments)

Comments CVPR 2024 Workshop on What is Next in Multimodal Foundation Models

IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval

Hui Chen, Guiguang Ding, Xudong Liu, Zijia Lin, Ji Liu, Jungong Han

专题命中 跨模态检索 ：cross-modal(title)；image-text(title)；分类 cs.CV

Comments 9 pages; Accepted by CVPR2020

MKG-CARE: Case-Aware Reasoning with Multimodal Knowledge Graphs for Explainable Medical Image Diagnosis

基于多模态知识图谱和可靠性引导精化的病例感知医学图像分类

Yiming Xu, Yixuan Liu, Yuhang Zhang, Ling Zheng, Yihan Wang, Qi Song

机构 * University of Science and Technology of China（科学技术大学） 纠错

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

AI总结 提出一种基于多模态知识图谱的病例感知推理框架，通过构建结构化诊断记忆、自适应检索相似病例、知识传播与注入机制以及置信度校准的决策精化方案，提升医学图像分类的性能和可解释性。

VERDICT: Training-Free Step-Wise Verification of Multimodal Reasoning via Disagreement-Aware Consensus

VERDICT：基于分歧感知共识的多模态推理无训练逐步验证方法

Rohit Sinha, Kunal Tilaganji, Tanuja Ganu, Nagarajan Natarajan, Amit Sharma, Vineeth Balasubramanian

机构 * Indian Institute of Technology, Hyderabad（印度理工学院海得拉巴分校） ； Microsoft Research（微软研究院） 纠错

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

AI总结 该研究针对多模态大语言模型推理链易出错的问题，提出无训练的VERDICT方法，利用跨模态分歧的闭式解计算共识评分，在六个基准上提升最高达5.95%，性能接近需大量监督的特定领域评论家。

Comments European Conference on Computer Vision 2026

Conan-embedding-v3: Fusing Modality-Specific Models for Omni-Modal Embedding

Conan-embedding-v3: 融合模态特定模型实现全模态嵌入

Shiyu Li, Zhiyuan Hu, Yifan Wang, Peiming Li, Zheng Wei, Yang Tang

机构 * Tencent（腾讯） 纠错

专题命中 跨模态检索 ：omni-modal(title,abstract)；multi-modal(abstract)；分类 cs.AI、cs.MM

AI总结 提出解耦-融合-恢复框架，通过独立训练模态专家并融合任务向量，再使用投影器恢复和平衡多模态重演解决投影器漂移问题，实现单一骨干网络支持文本、图像、视频、文档和音频检索。

MultiMem: Measuring and Mitigating Memorization in Multi-Modal Contrastive Learning

MultiMem: 多模态对比学习中的记忆化测量与缓解

Wenhao Wang, Franziska Boenisch, Michael Backes, Adam Dziedzic

机构 * CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心） 纠错

专题命中 跨模态检索 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

AI总结 提出首个多模态对比学习记忆化度量MultiMem，发现跨模态语义错位是主要驱动因素，文本模态影响最大，并通过跨模态增强有效降低记忆化并提升性能。

Comments Accepted at The 19th European Conference on Computer Vision (ECCV), 2026

Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Heads

多模态大语言模型中通过CoRe头的功能稀疏性机制洞察

Ruoxi Sun, Quantong Qiu, Juntao Li, Zecheng Tang, Yihang Lou, Min Zhang

机构 * Soochow University（苏州大学） ； Peking University（北京大学） 纠错

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI

AI总结 通过识别和分析CoRe头，揭示多模态大语言模型在跨模态检索中功能稀疏的结构特性，并验证其必要性及加速推理的潜力。

Reading, Not Thinking: Understanding and Bridging the Modality Gap When Text Becomes Pixels in Multimodal LLMs

阅读，而非思考：理解并弥合多模态大语言模型中文本变为像素时的模态差距

Kaiser Sun, Xiaochuang Yuan, Hongjun Liu, Chen Zhao, Cheng Zhang, Mark Dredze, Fan Bai

机构 * Johns Hopkins University（约翰霍普金斯大学） ； Amazon（亚马逊） ； New York University（纽约大学） ； Texas A&M University（德克萨斯大学） 纠错

专题命中 跨模态检索 ：multimodal(title,abstract)；MLLM(abstract_cn)；分类 cs.CV、cs.CL

AI总结 本文系统诊断多模态大语言模型在处理图像文本时的模态差距，发现其源于模型推理意愿不足而非感知失败，并提出一种轻量级自蒸馏方法有效弥合该差距。

xModel-KD: Cross-modal Knowledge Distillation for 3D Scene Perception using LiDAR

xModel-KD：基于LiDAR的3D场景感知跨模态知识蒸馏

Thenukan Pathmanathan, Kanchan Keisham, Thangarajah Akilan

机构 * Dept. of Computer Science Lakehead University Thunder Bay, Canada ； School of Computer Science Engg. \& Info. Systems Vellore Institute of Technology Tamil Nadu, India ； Dept. of Software Engg. Lakehead University Thunder Bay, Canada 纠错

专题命中 跨模态检索 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

AI总结 提出跨模态知识蒸馏框架xModel-KD，通过对比学习对齐2D图像纹理与3D点云几何特征，在无额外标注下提升LiDAR点云分割性能。

Comments 3 figures, and 5 tables

Contrastive Multi-Modal Hypergraph Reasoning for 3D Crowd Mesh Recovery

对比多模态超图推理用于3D人群网格重建

Minghao Sun, Chongyang Xu, Yitao Xie, Buzhen Huang, Kun Li

机构 * Tianjin University（天津大学） ； Nanyang Technological University（南洋理工大学） ； Sichuan University（四川大学） 纠错

专题命中 跨模态检索 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

AI总结 本文提出对比多模态超图推理方法，结合语义、几何和姿态线索，解决人群重建中的遮挡和深度模糊问题，通过超图建模高阶人群动态，提升特征融合与跨模态正交性。

Comments ICME 2026

Grounded Multimodal Retrieval-Augmented Drafting of Radiology Impressions Using Case-Based Similarity Search

基于案例相似性搜索的医学影像印象 grounded 多模态检索增强草稿生成

Himadri S Samanta

机构 * Independent AI Researcher（独立AI研究员） 纠错

专题命中 跨模态检索 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

AI总结 本文提出一种多模态检索增强生成系统，结合对比图像-文本嵌入、基于案例的相似性检索和引用约束草稿生成，以生成具有临床依据的医学影像印象。

Comments 15 pages, 4 figures, 3 tables

MiMIC: Mitigating Visual Modality Collapse in Universal Multimodal Retrieval While Avoiding Semantic Misalignment

MiMIC: 缓解通用多模态检索中的视觉模态崩溃并避免语义错位

Juan Li, Chuanghao Ding, Xujie Zhang, Cam-Tu Nguyen

机构 * State Key Laboratory for Novel Software Technology（新型软件技术国家重点实验室） ； School of Artificial Intelligence（人工智能学院） 纠错

专题命中 跨模态检索 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

AI总结 MiMIC通过融合-解码器架构和鲁棒训练方法，解决通用多模态检索中视觉模态崩溃和语义错位问题，在WebQA+和EVQA+数据集上优于早融合和晚融合基线。

Indexing Multimodal Language Models for Large-scale Image Retrieval

多模态大语言模型的索引用于大规模图像检索

Bahey Tharwat, Giorgos Kordopatis-Zilos, Pavel Suma, Ian Reid, Giorgos Tolias

机构 * Mohamed bin Zayed University of Artificial Intelligence（莫扎德·本·扎耶德人工智能大学） ； VRG, FEE, Czech Technical University in Prague（布拉格捷克技术大学VRG学院） 纠错

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

AI总结 本文探讨了多模态大语言模型作为无训练相似性估计器在实例级图像到图像检索中的应用，通过提示配对图像并转换下一个词的概率为相似性分数，实现了零样本重排序，展示了其在大规模检索中的优势。