多模态大模型

VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents

Rui Meng, Ziyan Jiang, Ye Liu, Mingyi Su, Xinyi Yang, Yuepeng Fu, Can Qin, Zeyuan Chen, Ran Xu, Caiming Xiong, Yingbo Zhou, Wenhu Chen, Semih Yavuz

机构 * Salesforce Research（Salesforce研究机构） ； UC Santa Barbara（加州大学圣巴巴拉分校） ； University of Waterloo（滑铁卢大学） ； Tsinghua University（清华大学） 纠错

专题命中 跨模态检索 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

Comments Technical Report

Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval

Fanheng Kong, Jingyuan Zhang, Yahui Liu, Hongzhi Zhang, Shi Feng, Xiaocui Yang, Daling Wang, Yu Tian, Victoria W., Fuzheng Zhang, Guorui Zhou

机构 * Northeastern University（东北大学） ； Kuaishou Technology（快手科技） 纠错

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments 26 pages, project page: https://friedrichor.github.io/projects/UNITE

ChatEXAONEPath: An Expert-level Multimodal Large Language Model for Histopathology Using Whole Slide Images

Sangwook Kim, Soonyoung Lee, Jongseong Jang

专题命中 跨模态检索 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL

IDMR: Towards Instance-Driven Precise Visual Correspondence in Multimodal Retrieval

Bangwei Liu, Yicheng Bao, Shaohui Lin, Xuhong Wang, Xin Tan, Yingchun Wang, Yuan Xie, Chaochao Lu

专题命中 跨模态检索 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Multi-modal Vision Pre-training for Medical Image Analysis

Shaohao Rui, Lingzhi Chen, Zhenyu Tang, Lilong Wang, Mianxin Liu, Shaoting Zhang, Xiaosong Wang

专题命中 跨模态检索 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Journal ref CVPR2025

CSA: Data-efficient Mapping of Unimodal Features to Multimodal Features

Po-han Li, Sandeep P. Chinchali, Ufuk Topcu

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Journal ref Published at ICLR 2025

TI-JEPA: An Innovative Energy-based Joint Embedding Strategy for Text-Image Multimodal Systems

Khang H. N. Vo, Duc P. T. Nguyen, Thong Nguyen, Tho T. Quan

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

MV-CLAM: Multi-View Molecular Interpretation with Cross-Modal Projection via Language Model

Sumin Ha, Jun Hyeong Kim, Yinhua Piao, Sun Kim

专题命中 跨模态检索 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CL、cs.AI

ClusMFL: A Cluster-Enhanced Framework for Modality-Incomplete Multimodal Federated Learning in Brain Imaging Analysis

Xinpeng Wang, Rong Zhou, Han Xie, Xiaoying Tang, Lifang He, Carl Yang

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Towards Identity-Aware Cross-Modal Retrieval: a Dataset and a Baseline

Nicola Messina, Lucia Vadicamo, Leo Maltese, Claudio Gennaro

专题命中 跨模态检索 ：cross-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM

Comments Accepted as full paper at ECIR 2025

Deep Reversible Consistency Learning for Cross-modal Retrieval

Ruitao Pu, Yang Qin, Dezhong Peng, Xiaomin Song, Huiming Zheng

专题命中 跨模态检索 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM

Multimodal Multihop Source Retrieval for Web Question Answering

Navya Yarrabelly, Saloni Mittal

专题命中 跨模态检索 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CL、cs.AI

Comments arXiv admin note: text overlap with arXiv:2010.03604 by other authors

CL3DOR: Contrastive Learning for 3D Large Multimodal Models via Odds Ratio on High-Resolution Point Clouds

Keonwoo Kim, Yeongjae Cho, Taebaek Hwang, Minsoo Jo, Sangdo Han

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Robust Self-Paced Hashing for Cross-Modal Retrieval with Noisy Labels

Ruitao Pu, Yuan Sun, Yang Qin, Zhenwen Ren, Xiaomin Song, Huiming Zheng, Dezhong Peng

专题命中 跨模态检索 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM

Comments 9 pages, AAAI 25 conference

Multi-Head Attention Driven Dynamic Visual-Semantic Embedding for Enhanced Image-Text Matching

Wenjing Chen

专题命中 跨模态检索 ：image-text(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL

Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval

Rui Cai, Zhiyu Dong, Jianfeng Dong, Xun Wang

专题命中 跨模态检索 ：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Accepted by the 39th AAAI Conference on Artificial Intelligence (AAAI-25)

Multi-level Matching Network for Multimodal Entity Linking

Zhiwei Hu, Víctor Gutiérrez-Basulto, Ru Li, Jeff Z. Pan

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted at KDD'25

Multimodal Whole Slide Foundation Model for Pathology

Tong Ding, Sophia J. Wagner, Andrew H. Song, Richard J. Chen, Ming Y. Lu, Andrew Zhang, Anurag J. Vaidya, Guillaume Jaume, Muhammad Shaban, Ahrong Kim, Drew F. K. Williamson, Bowen Chen, Cristina Almagro-Perez, Paul Doucet, Sharifa Sahai, Chengkuan Chen, Daisuke Komura, Akihiro Kawabe, Shumpei Ishikawa, Georg Gerber, Tingying Peng, Long Phi Le, Faisal Mahmood

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments The code is accessible at https://github.com/mahmoodlab/TITAN

FLEX-CLIP: Feature-Level GEneration Network Enhanced CLIP for X-shot Cross-modal Retrieval

Jingyou Xie, Jiayi Kuang, Zhenzhou Lin, Jiarui Ouyang, Zishuo Zhao, Ying Shen

专题命中 跨模态检索 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL

Any2Any: Incomplete Multimodal Retrieval with Conformal Prediction

Po-han Li, Yunhao Yang, Mohammad Omama, Sandeep Chinchali, Ufuk Topcu

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

MoRE: Multi-Modal Contrastive Pre-training with Transformers on X-Rays, ECGs, and Diagnostic Report

Samrajya Thapa, Koushik Howlader, Subhankar Bhattacharjee, Wei le

专题命中 跨模态检索 ：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments 10 pages, 5 figures, 9 tables. Supplementary detail in Appendix. Code made available in Github for reproducibility

BadCM: Invisible Backdoor Attack Against Cross-Modal Learning

Zheng Zhang, Xu Yuan, Lei Zhu, Jingkuan Song, Liqiang Nie

专题命中 跨模态检索 ：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.MM

Journal ref IEEE Transactions on Image Processing, vol. 33, pp. 2558-2571, 2024

From Latent to Engine Manifolds: Analyzing ImageBind's Multimodal Embedding Space

Andrew Hamara, Pablo Rivas

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments The 26th International Conference on Artificial Intelligence (ICAI'24)

OrthoDoc: Multimodal Large Language Model for Assisting Diagnosis in Computed Tomography

Youzhu Jin, Yichen Zhang

专题命中 跨模态检索 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments 8 pages, 1 figure

DM2RM: Dual-Mode Multimodal Ranking for Target Objects and Receptacles Based on Open-Vocabulary Instructions

Ryosuke Korekata, Kanta Kaneda, Shunya Nagashima, Yuto Imai, Komei Sugiura

专题命中 跨模态检索 ：multimodal(title,abstract)；multimodal foundation model(abstract)；分类 cs.CV、cs.CL

Spatiotemporal Graph Guided Multi-modal Network for Livestreaming Product Retrieval

Xiaowan Hu, Yiyi Chen, Yan Li, Minquan Wang, Haoqian Wang, Quan Chen, Han Li, Peng Jiang

专题命中 跨模态检索 ：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.MM

Comments 16 pages, 12 figures

Contrast then Memorize: Semantic Neighbor Retrieval-Enhanced Inductive Multimodal Knowledge Graph Completion

Yu Zhao, Ying Zhang, Baohang Zhou, Xinying Qian, Kehui Song, Xiangrui Cai

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL、cs.MM

Comments Accepted by SIGIR 2024

Improving the Consistency in Cross-Lingual Cross-Modal Retrieval with 1-to-K Contrastive Learning

Zhijie Nie, Richong Zhang, Zhangchi Feng, Hailang Huang, Xudong Liu

专题命中 跨模态检索 ：cross-modal(title,abstract)；image-text(abstract)；分类 cs.AI、cs.MM

Comments Accepted by KDD 2024 Research Track

Linking Representations with Multimodal Contrastive Learning

Abhishek Arora, Xinmei Yang, Shao-Yu Jheng, Melissa Dell

专题命中 跨模态检索 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval

Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, Yongping Xiong

专题命中 跨模态检索 ：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Accepted to ACL 2024 main conference