多模态大模型 - arXivDaily 专题

2505.24073 2025-08-27 cs.AI cs.CL cs.CV 82%

mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation

Chan-Wei Hu, Yueqi Wang, Shuo Xing, Chia-Ju Chen, Suofei Feng, Ryan Rossi, Zhengzhong Tu

机构 * Texas A&M University（德克萨斯农工大学）； University of California, Berkeley（加州大学伯克利分校）； University of Texas at Austin（德克萨斯大学奥斯汀分校）； Stanford University（斯坦福大学）； Adobe Research（Adobe研究院）

专题命中跨模态检索：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 16 pages

2508.11673 2025-08-19 cs.LG cs.AI cs.CV cs.MM 82%

Contrastive Regularization over LoRA for Multimodal Biomedical Image Incremental Learning

Haojie Zhang, Yixiong Liang, Hulin Kuang, Lihui Cen, Zhe Qu, Yigang Cen, Min Zeng, Shichao Kan

机构 * School of Computer Science and Engineering, Central South University（计算机科学与工程学院，中南大学）； School of Automation, Central South University（自动化学院，中南大学）； School of Computer Science and Technology, Beijing Jiaotong University（计算机科学与技术学院，北京交通大学）

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 10 pages, 3 figures, submitted to ACM Multimedia 2025

2508.11141 2025-08-18 cs.CV cs.AI cs.CL 82%

A Cross-Modal Rumor Detection Scheme via Contrastive Learning by Exploring Text and Image internal Correlations

Bin Ma, Yifei Zhang, Yongjin Xian, Qi Li, Linna Zhou, Gongxun Miao

专题命中跨模态检索：cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2508.02538 2025-08-05 cs.IR 82%

Hubness Reduction with Dual Bank Sinkhorn Normalization for Cross-Modal Retrieval

Zhengxin Pan, Haishuai Wang, Fangyu Wu, Peng Zhang, Jiajun Bu

专题命中跨模态检索：cross-modal(title,abstract)；image-text(abstract)

Comments ACMMM 2025

2507.18009 2025-07-25 cs.CV cs.AI cs.CL 82%

GRR-CoCa: Leveraging LLM Mechanisms in Multimodal Model Architectures

Jake R. Patock, Nicole Catherine Lewis, Kevin McCoy, Christina Gomez, Canling Chen, Lorenzo Luzi

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 12 pages, 2 figures

2506.19329 2025-06-25 cs.LG 82%

Contrastive Cross-Modal Learning for Infusing Chest X-ray Knowledge into ECGs

Vineet Punyamoorty, Aditya Malusare, Vaneet Aggarwal

专题命中跨模态检索：cross-modal(title,abstract)；multimodal(abstract)

2504.03184 2025-06-25 cs.IR 82%

Answering Multimodal Exclusion Queries with Lightweight Sparse Disentangled Representations

Prachi J, Sumit Bhatia, Srikanta Bedathur

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)

Comments In Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR)

2506.12364 2025-06-24 cs.AI cs.CL cs.CV 82%

MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrieval

Mingjun Xu, Jinhan Dong, Jue Hou, Zehui Wang, Sihang Li, Zhifeng Gao, Renxin Zhong, Hengxing Cai

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2311.06456 2025-05-15 cs.LG 82%

Advancing Drug Discovery with Enhanced Chemical Understanding via Asymmetric Contrastive Multimodal Learning

Yifei Wang, Yunrui Li, Lin Liu, Pengyu Hong, Hao Xu

机构 * Brandeis University（布兰迪斯大学）； Stanford University（斯坦福大学）； Northeastern University（东北大学）； Harvard Medical School（哈佛医学院）

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)

Comments 51 pages, 19 figures, 11 tables

2412.13509 2025-03-26 cs.HC 82%

Visualizing the Invisible: A Generative AR System for Intuitive Multi-Modal Sensor Data Presentation

Yunqi Guo, Kaiyuan Hou, Heming Fu, Hongkai Chen, Zhenyu Yan, Guoliang Xing, Xiaofan Jiang

专题命中跨模态检索：multi-modal(title,abstract)；cross-modal(abstract)

2407.19711 2025-03-24 cs.SE 82%

TVDiag: A Task-oriented and View-invariant Failure Diagnosis Framework with Multimodal Data

Shuaiyu Xie, Jian Wang, Hanbin He, Zhihao Wang, Yuqi Zhao, Neng Zhang, Bing Li

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)

Comments 32 pages

2503.16064 2025-03-21 cs.CV cs.AI cs.IR cs.MM 82%

PromptHash: Affinity-Prompted Collaborative Cross-Modal Learning for Adaptive Hashing Retrieval

Qiang Zou, Shuli Cheng, Jiayi Chen

专题命中跨模态检索：cross-modal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by CVPR2025

2503.04252 2025-03-07 cs.DB cs.LG 82%

RCRank: Multimodal Ranking of Root Causes of Slow Queries in Cloud Database Systems

Biao Ouyang, Yingying Zhang, Hanyin Cheng, Yang Shu, Chenjuan Guo, Bin Yang, Qingsong Wen, Lunting Fan, Christian S. Jensen

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)

Comments Accepted by VLDB 2025

2411.12126 2025-03-06 cs.LG 82%

MMBind: Unleashing the Potential of Distributed and Heterogeneous Data for Multimodal Learning in IoT

Xiaomin Ouyang, Jason Wu, Tomoyoshi Kimura, Yihan Lin, Gunjan Verma, Tarek Abdelzaher, Mani Srivastava

专题命中跨模态检索：multimodal(title,abstract)；multimodal foundation model(abstract)

2409.07402 2025-03-06 cs.LG cs.AI cs.CL cs.CV 82%

What to align in multimodal contrastive learning?

Benoit Dufumier, Javiera Castillo-Navarro, Devis Tuia, Jean-Philippe Thiran

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ICLR 2025, 25 pages

2502.18763 2025-02-27 cs.IT math.IT 82%

CommGPT: A Graph and Retrieval-Augmented Multimodal Communication Foundation Model

Feibo Jiang, Wanyun Zhu, Li Dong, Kezhi Wang, Kun Yang, Cunhua Pan, Octavia A. Dobre

专题命中跨模态检索：multimodal(title,abstract)；multimodal foundation model(abstract)

2501.17799 2025-02-18 cs.HC cs.IR 82%

Leveraging Multimodal LLM for Inspirational User Interface Search

Seokhyeon Park, Yumin Song, Soohyun Lee, Jaeyoung Kim, Jinwook Seo

专题命中跨模态检索：multimodal(title,abstract)；MLLM(abstract)

Comments In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '25)

2502.00653 2025-02-04 cs.CR 82%

Towards Robust Multimodal Large Language Models Against Jailbreak Attacks

Ziyi Yin, Yuanpu Cao, Han Liu, Ting Wang, Jinghui Chen, Fenhlong Ma

专题命中跨模态检索：multimodal(title,abstract)；MLLM(abstract)

2410.05160 2025-01-03 cs.CV cs.AI cs.CL 82%

VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks

Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, Wenhu Chen

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Technical Report

2412.14835 2024-12-20 cs.CL cs.AI cs.CV cs.IR 82%

Progressive Multimodal Reasoning via Active Retrieval

Guanting Dong, Chenghao Zhang, Mengjie Deng, Yutao Zhu, Zhicheng Dou, Ji-Rong Wen

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Working in progress

2409.12728 2024-12-19 q-bio.GN cs.LG 82%

PRAGA: Prototype-aware Graph Adaptive Aggregation for Spatial Multi-modal Omics Analysis

Xinlei Huang, Zhiqi Ma, Dian Meng, Yanran Liu, Shiwei Ruan, Qingqiang Sun, Xubin Zheng, Ziyue Qiao

专题命中跨模态检索：multi-modal(title,abstract)；cross-modal(abstract)

Comments Accepted by AAAl2025; full version including appendix

2411.16096 2024-11-26 cs.CV cs.AI cs.MM 82%

ENCLIP: Ensembling and Clustering-Based Contrastive Language-Image Pretraining for Fashion Multimodal Search with Limited Data and Low-Quality Images

Prithviraj Purushottam Naik, Rohit Agarwal

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

2411.04952 2024-11-08 cs.CV cs.AI cs.CL 82%

M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding

Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, Mohit Bansal

专题命中跨模态检索：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project webpage: https://m3docrag.github.io

2410.20482 2024-10-29 cs.CL cs.AI cs.CV 82%

What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration

Libo Qin, Qiguang Chen, Hao Fei, Zhi Chen, Min Li, Wanxiang Che

专题命中跨模态检索：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at NeurIPS 2024

2409.12295 2024-09-20 cs.LG stat.ML 82%

SANE: Strategic Autonomous Non-Smooth Exploration for Multiple Optima Discovery in Multi-modal and Non-differentiable Black-box Functions

Arpan Biswas, Rama Vasudevan, Rohit Pant, Ichiro Takeuchi, Hiroshi Funakubo, Yongtao Liu

专题命中跨模态检索：multi-modal(title,abstract)；multimodal(abstract)

Comments 25 pages, 7 figures in main text, 2 figures in Supp Mat

2408.00491 2024-08-02 cs.CL cs.CV cs.MM 82%

GalleryGPT: Analyzing Paintings with Large Multimodal Models

Yi Bin, Wenhao Shi, Yujuan Ding, Zhiqiang Hu, Zheng Wang, Yang Yang, See-Kiong Ng, Heng Tao Shen

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted as Oral Presentation at ACM Multimedia 2024

2407.20709 2024-07-31 cs.RO 82%

A Case Study on Visual-Audio-Tactile Cross-Modal Retrieval

Jagoda Wojcik, Jiaqi Jiang, Jiacheng Wu, Shan Luo

专题命中跨模态检索：cross-modal(title,abstract)；multi-modal(abstract)

Comments 7 pages, 6 figures, accepted to the 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2024)

2407.06219 2024-07-10 cs.NE 82%

In Search of Excellence: SHOA as a Competitive Shrike Optimization Algorithm for Multimodal Problems

Hanan K. AbdulKarim, Tarik A. Rashid

专题命中跨模态检索：multimodal(title,abstract)；multi-modal(abstract)

Comments 25 pages

2309.05423 2024-06-12 eess.AS cs.AI cs.CL cs.SD 82%

Multi-Modal Automatic Prosody Annotation with Contrastive Pretraining of SSWP

Jinzuomu Zhong, Yang Li, Hui Huang, Korin Richmond, Jie Liu, Zhiba Su, Jing Guo, Benlai Tang, Fengjie Zhu

专题命中跨模态检索：multi-modal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

2405.17777 2024-05-29 cs.IR 82%

RREH: Reconstruction Relations Embedded Hashing for Semi-Paired Cross-Modal Retrieval

Jianzong Wang, Haoxiang Shi, Kaiyi Luo, Xulong Zhang, Ning Cheng, Jing Xiao

专题命中跨模态检索：cross-modal(title,abstract)；multi-modal(abstract)

Comments Accepted by the 20th International Conference on Intelligent Computing (ICIC 2024)

1. 跨模态检索 3463 篇

mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation

Contrastive Regularization over LoRA for Multimodal Biomedical Image Incremental Learning

A Cross-Modal Rumor Detection Scheme via Contrastive Learning by Exploring Text and Image internal Correlations

Hubness Reduction with Dual Bank Sinkhorn Normalization for Cross-Modal Retrieval

GRR-CoCa: Leveraging LLM Mechanisms in Multimodal Model Architectures

Contrastive Cross-Modal Learning for Infusing Chest X-ray Knowledge into ECGs

Answering Multimodal Exclusion Queries with Lightweight Sparse Disentangled Representations

MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrieval

Advancing Drug Discovery with Enhanced Chemical Understanding via Asymmetric Contrastive Multimodal Learning

Visualizing the Invisible: A Generative AR System for Intuitive Multi-Modal Sensor Data Presentation

TVDiag: A Task-oriented and View-invariant Failure Diagnosis Framework with Multimodal Data

PromptHash: Affinity-Prompted Collaborative Cross-Modal Learning for Adaptive Hashing Retrieval

RCRank: Multimodal Ranking of Root Causes of Slow Queries in Cloud Database Systems

MMBind: Unleashing the Potential of Distributed and Heterogeneous Data for Multimodal Learning in IoT

What to align in multimodal contrastive learning?

CommGPT: A Graph and Retrieval-Augmented Multimodal Communication Foundation Model

Leveraging Multimodal LLM for Inspirational User Interface Search

Towards Robust Multimodal Large Language Models Against Jailbreak Attacks

VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks

Progressive Multimodal Reasoning via Active Retrieval

PRAGA: Prototype-aware Graph Adaptive Aggregation for Spatial Multi-modal Omics Analysis

ENCLIP: Ensembling and Clustering-Based Contrastive Language-Image Pretraining for Fashion Multimodal Search with Limited Data and Low-Quality Images

M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding

What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration

SANE: Strategic Autonomous Non-Smooth Exploration for Multiple Optima Discovery in Multi-modal and Non-differentiable Black-box Functions

GalleryGPT: Analyzing Paintings with Large Multimodal Models

A Case Study on Visual-Audio-Tactile Cross-Modal Retrieval

In Search of Excellence: SHOA as a Competitive Shrike Optimization Algorithm for Multimodal Problems

Multi-Modal Automatic Prosody Annotation with Contrastive Pretraining of SSWP

RREH: Reconstruction Relations Embedded Hashing for Semi-Paired Cross-Modal Retrieval