多模态大模型 - arXivDaily 专题

2106.12570 2022-12-19 cs.LG cs.CV 83%

Learning Multimodal VAEs through Mutual Supervision

Tom Joy, Yuge Shi, Philip H. S. Torr, Tom Rainforth, Sebastian M. Schmon, N. Siddharth

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

2203.14713 2022-11-21 cs.IR cs.AI 83%

Image-text Retrieval: A Survey on Recent Research and Development

Min Cao, Shiping Li, Juntao Li, Liqiang Nie, Min Zhang

专题命中图文多模态：image-text(title,abstract)；cross-modal(abstract)；分类 cs.AI

Comments Accpted by IJCAI'2022 survey track

2210.13188 2022-10-25 cs.CV 83%

Dissecting Deep Metric Learning Losses for Image-Text Retrieval

Hong Xuan, Xi Chen

专题命中图文多模态：image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments arXiv admin note: text overlap with arXiv:2201.11307

Journal ref WACV2023

2205.14204 2022-10-24 cs.CV 83%

Multimodal Masked Autoencoders Learn Transferable Representations

Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, Pieter Abbeel

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

2111.11430 2022-07-20 cs.CV 83%

Class-agnostic Object Detection with Multi-modal Transformer

Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, Ming-Hsuan Yang

专题命中图文多模态：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted at ECCV 2022

2201.06686 2022-06-07 cs.CV 83%

Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching

Hengcan Shi, Munawar Hayat, Jianfei Cai

专题命中图文多模态：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments 9 pages, 7 figures

2204.04014 2022-05-06 cs.CV cs.NE 83%

Multimodal Quasi-AutoRegression: Forecasting the visual popularity of new fashion products

Stefanos I. Papadopoulos, Christos Koutlis, Symeon Papadopoulos, Ioannis Kompatsiaris

专题命中图文多模态：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments 14 pages, 4 figures

2204.05533 2022-04-28 cs.CL cs.SI 83%

How does fake news use a thumbnail? CLIP-based Multimodal Detection on the Unrepresentative News Image

Hyewon Choi, Yejun Yoon, Seunghyun Yoon, Kunwoo Park

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CL

Comments 9 pages, 8 figures including appendix figure, 2 tables. Published in Findings of ACL workshop, CONSTRAINT 2022 (Long paper). The manuscript is slightly revised after the camera ready version

2204.04588 2022-04-12 cs.CV cs.LG 83%

Robust Cross-Modal Representation Learning with Progressive Self-Distillation

Alex Andonian, Shixing Chen, Raffay Hamid

专题命中图文多模态：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted to CVPR 2022

2202.06076 2022-02-15 cs.CV cs.LG 83%

Indication as Prior Knowledge for Multimodal Disease Classification in Chest Radiographs with Transformers

Grzegorz Jacenków, Alison Q. O'Neil, Sotirios A. Tsaftaris

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted at the IEEE International Symposium on Biomedical Imaging (ISBI) 2022 as an oral presentation

2108.08217 2021-08-19 cs.CV cs.AI cs.CL cs.LG cs.MM 83%

X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics

Yehao Li, Yingwei Pan, Jingwen Chen, Ting Yao, Tao Mei

专题命中图文多模态：cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by 2021 ACMMM Open Source Software Competition. Source code: https://github.com/YehLi/xmodaler

2107.14596 2021-08-02 cs.CL 83%

Multi-stage Pre-training over Simplified Multimodal Pre-training Models

Tongtong Liu, Fangxiang Feng, Xiaojie Wang

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CL

Comments 10 pages, 4 figures

2107.11970 2021-07-27 cs.CV 83%

Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph

Wentian Zhao, Yao Hu, Heda Wang, Xinxiao Wu, Jiebo Luo

专题命中图文多模态：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2107.10300 2021-07-23 cs.CV cs.AI cs.CL cs.LG cs.MM eess.IV 83%

iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretability

Aman Chadha, Vinija Jain

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 12 pages, 1 figure, 7 tables

2106.06509 2021-06-14 cs.CV 83%

Step-Wise Hierarchical Alignment Network for Image-Text Matching

Zhong Ji, Kexin Chen, Haoran Wang

专题命中图文多模态：image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by IJCAI 2021

2104.12471 2021-04-27 cs.CV cs.AI cs.CL cs.IR cs.MM 83%

Contextualized Keyword Representations for Multi-modal Retinal Image Captioning

Jia-Hong Huang, Ting-Wei Wu, Marcel Worring

专题命中图文多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments This paper is accepted by ACM International Conference on Multimedia Retrieval (ICMR), 2021

2104.10406 2021-04-22 cs.CV 83%

Discrete-continuous Action Space Policy Gradient-based Attention for Image-Text Matching

Shiyang Yan, Li Yu, Yuan Xie

专题命中图文多模态：image-text(title,abstract)；multi-modal(abstract)；分类 cs.CV

2006.09310 2020-06-17 cs.CV cs.LG 83%

Deep Multimodal Transfer-Learned Regression in Data-Poor Domains

Levi McClenny, Mulugeta Haile, Vahid Attari, Brian Sadler, Ulisses Braga-Neto, Raymundo Arroyave

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

1908.06066 2019-12-04 cs.CV 83%

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, Ming Zhou

专题命中图文多模态：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments accepted by AAAI-2020. arXiv admin note: text overlap with arXiv:1909.11740 by other authors

1907.12490 2019-07-30 cs.IR cs.MM 83%

Deep Cross-Modal Hashing with Hashing Functions and Unified Hash Codes Jointly Learning

Rong-Cheng Tu, Xian-Ling Mao, Bing Ma, Yong Hu, Tan Yan, Wei Wei, Heyan Huang

专题命中图文多模态：cross-modal(title,abstract)；image-text(abstract)；分类 cs.MM

1906.09610 2019-06-25 cs.CV 83%

Improving Description-based Person Re-identification by Multi-granularity Image-text Alignments

Kai Niu, Yan Huang, Wanli Ouyang, Liang Wang

专题命中图文多模态：image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV

1904.06286 2019-04-17 cs.CL 83%

CITE: A Corpus of Image-Text Discourse Relations

Malihe Alikhani, Sreyasi Nag Chowdhury, Gerard de Melo, Matthew Stone

专题命中图文多模态：image-text(title,abstract)；multimodal(abstract)；分类 cs.CL

Comments 7 pages

Journal ref 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics

1809.04144 2018-09-13 cs.CV 83%

End-to-end Image Captioning Exploits Multimodal Distributional Similarity

Pranava Madhyastha, Josiah Wang, Lucia Specia

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Published in BMVC 2018

1611.00471 2017-03-22 cs.CV 83%

Dual Attention Networks for Multimodal Reasoning and Matching

Hyeonseob Nam, Jung-Woo Ha, Jeonghee Kim

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

1411.4738 2015-12-18 cs.MM cs.IR cs.LG 83%

Cross-Modal Similarity Learning : A Low Rank Bilinear Formulation

Cuicui Kang, Shengcai Liao, Yonghao He, Jian Wang, Wenjia Niu, Shiming Xiang, Chunhong Pan

专题命中图文多模态：cross-modal(title,abstract)；image-text(abstract)；分类 cs.MM

2504.21375 2025-05-01 cs.LG 83%

Synergy-CLIP: Extending CLIP with Multi-modal Integration for Robust Representation Learning

Sangyeon Cho, Jangyeong Jeon, Mingi Kim, Junyeong Kim

机构 * Department of Artificial Intelligence, Chung-Ang University, Seoul 06974, Republic of Korea（人工智能系，成均馆大学，韩国首尔）

专题命中图文多模态：multi-modal(title,abstract)；image-text(abstract)

Comments Multi-modal, Multi-modal Representation Learning, Missing Modality, Missing Modality Reconstruction, Speech and Multi-modality, Vision and Language

2510.22665 2026-08-11 cs.CV cs.AI 版本更新 82%

SARVLM: A Vision Language Foundation Model for Semantic Understanding in SAR Imagery

SARVLM：一种用于SAR图像语义理解的视觉语言基础模型

Qiwei Ma, Xukun Lu, Wang Liu, Puhong Duan, Xudong Kang, Shutao Li

机构 * School of Artificial Intelligence and Robotics, Hunan University（湖南大学人工智能与机器人学院）； Yuelushan Center for Industrial Innovation（岳麓山创新中心）； School of Medical Information Engineering, Jining Medical University（济南医学院医学信息工程学院）

专题命中图文多模态：multimodal(abstract)；multi-modal(abstract)；cross-modal(abstract)；image-text(abstract)

AI总结本文提出SARVLM，一种专为SAR图像设计的视觉语言基础模型，通过多阶段领域迁移策略和集成策略提升跨场景泛化能力，在多个基准测试中表现优异。

Comments 13 pages, 13 figures

2208.10442 2022-09-01 cs.CV cs.CL 82%

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, Furu Wei

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)；multimodal foundation model(abstract)

Comments 18 pages

2608.11681 2026-08-13 cs.CV cs.AI cs.MM 新提交 82%

Learning from Multimodal Pseudo-Labels for Robust Open-Vocabulary Instance and Panoptic Segmentation

学习多模态伪标签以实现鲁棒的开放词汇实例和全景分割

Duy Tran Thanh, Yeejin Lee, Byeongkeun Kang

机构 * Seoul National University of Science and Technology（首尔科技大学）； Chung-Ang University（中央大学）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

AI总结该研究针对开放词汇实例分割与开放集全景分割的痛点，提出多模态框架，结合预训练视觉语言模型生成伪标签并优化训练目标，在COCO数据集上取得优于现有SOTA的性能。

Comments 14 pages

Journal ref Neurocomputing, 2026

2605.16411 2026-08-12 cs.CV cs.AI cs.CL cs.DB cs.LG 版本更新 82%

Grounded Post-Training with Hard Examples for Reducing Hallucination in Multimodal Large Language Models

通过分布偏移下的分阶段偏好优化减少视觉-语言模型中的幻觉

Qinwu Xu

机构 * Meta AI

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结本文提出分阶段偏好优化框架，通过构建针对幻觉问题的数据集，提升视觉-语言模型的 grounded reasoning，减少幻觉并提高响应信息量。

1. 图文多模态 4657 篇

Learning Multimodal VAEs through Mutual Supervision

Image-text Retrieval: A Survey on Recent Research and Development

Dissecting Deep Metric Learning Losses for Image-Text Retrieval

Multimodal Masked Autoencoders Learn Transferable Representations

Class-agnostic Object Detection with Multi-modal Transformer

Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching

Multimodal Quasi-AutoRegression: Forecasting the visual popularity of new fashion products

How does fake news use a thumbnail? CLIP-based Multimodal Detection on the Unrepresentative News Image

Robust Cross-Modal Representation Learning with Progressive Self-Distillation

Indication as Prior Knowledge for Multimodal Disease Classification in Chest Radiographs with Transformers

X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics

Multi-stage Pre-training over Simplified Multimodal Pre-training Models

Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph

iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretability

Step-Wise Hierarchical Alignment Network for Image-Text Matching

Contextualized Keyword Representations for Multi-modal Retinal Image Captioning

Discrete-continuous Action Space Policy Gradient-based Attention for Image-Text Matching

Deep Multimodal Transfer-Learned Regression in Data-Poor Domains

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

Deep Cross-Modal Hashing with Hashing Functions and Unified Hash Codes Jointly Learning

Improving Description-based Person Re-identification by Multi-granularity Image-text Alignments

CITE: A Corpus of Image-Text Discourse Relations

End-to-end Image Captioning Exploits Multimodal Distributional Similarity

Dual Attention Networks for Multimodal Reasoning and Matching

Cross-Modal Similarity Learning : A Low Rank Bilinear Formulation

Synergy-CLIP: Extending CLIP with Multi-modal Integration for Robust Representation Learning

SARVLM: A Vision Language Foundation Model for Semantic Understanding in SAR Imagery

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

Learning from Multimodal Pseudo-Labels for Robust Open-Vocabulary Instance and Panoptic Segmentation

Grounded Post-Training with Hard Examples for Reducing Hallucination in Multimodal Large Language Models