多模态大模型 - arXivDaily 专题

2105.05636 2023-01-06 cs.CV 70%

VL-NMS: Breaking Proposal Bottlenecks in Two-Stage Visual-Language Matching

Chenchi Zhang, Wenbo Ma, Jun Xiao, Hanwang Zhang, Jian Shao, Yueting Zhuang, Long Chen

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV

Comments arXiv admin note: substantial text overlap with arXiv:2009.01449

2212.07086 2023-01-05 cs.CV 70%

NLIP: Noise-robust Language-Image Pre-training

Runhui Huang, Yanxin Long, Jianhua Han, Hang Xu, Xiwen Liang, Chunjing Xu, Xiaodan Liang

专题命中图文多模态：cross-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments AAAI 2023

2212.08985 2022-12-20 cs.CV 70%

Efficient Image Captioning for Edge Devices

Ning Wang, Jiangrong Xie, Hang Luo, Qinglin Cheng, Jihao Wu, Mingbo Jia, Linlin Li

专题命中图文多模态：cross-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments To appear in AAAI 2023

2205.13125 2022-11-21 cs.CV 70%

Prompt-based Learning for Unpaired Image Captioning

Peipei Zhu, Xiao Wang, Lin Zhu, Zhenglong Sun, Weishi Zheng, Yaowei Wang, Changwen Chen

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV

2210.17322 2022-11-01 cs.CV 70%

Generative Negative Text Replay for Continual Vision-Language Pretraining

Shipeng Yan, Lanqing Hong, Hang Xu, Jianhua Han, Tinne Tuytelaars, Zhenguo Li, Xuming He

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments ECCV 2022

2210.10276 2022-10-20 cs.CV 70%

CLIP-Driven Fine-grained Text-Image Person Re-identification

Shuanglin Yan, Neng Dong, Liyan Zhang, Jinhui Tang

专题命中图文多模态：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV

2210.08901 2022-10-18 cs.CV 70%

Contrastive Language-Image Pre-Training with Knowledge Graphs

Xuran Pan, Tianzhu Ye, Dongchen Han, Shiji Song, Gao Huang

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted by NeurIPS2022

2208.02515 2022-09-20 cs.CV 70%

Fine-Grained Semantically Aligned Vision-Language Pre-Training

Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Qi Tian, Siliang Tang

专题命中图文多模态：cross-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted by NeurIPS 2022

2207.13162 2022-08-23 cs.CV cs.AI cs.CL cs.MM 70%

Retrieval-Augmented Transformer for Image Captioning

Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

专题命中图文多模态：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CBMI 2022

2208.09374 2022-08-22 cs.CV 70%

VLMAE: Vision-Language Masked Autoencoder

Sunan He, Taian Guo, Tao Dai, Ruizhi Qiao, Chen Wu, Xiujun Shu, Bo Ren

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments 12 pages, 7 figures

2112.03857 2022-06-20 cs.CV cs.AI cs.CL cs.LG cs.MM 70%

Grounded Language-Image Pre-training

Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, Jianfeng Gao

专题命中图文多模态：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2022; updated visualizations; fixed hyper-parameters in Appendix C.1

2206.06930 2022-06-15 cs.CV cs.AI cs.CL cs.MM 70%

Comprehending and Ordering Semantics for Image Captioning

Yehao Li, Yingwei Pan, Ting Yao, Tao Mei

专题命中图文多模态：cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2022; Code is publicly available at: https://github.com/YehLi/xmodaler/tree/master/configs/image_caption/cosnet

2108.05308 2022-02-03 cs.CV cs.LG 70%

A Better Loss for Visual-Textual Grounding

Davide Rigoni, Luciano Serafini, Alessandro Sperduti

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV

2111.10023 2021-11-22 cs.CV 70%

UFO: A UniFied TransfOrmer for Vision-Language Representation Learning

Jianfeng Wang, Xiaowei Hu, Zhe Gan, Zhengyuan Yang, Xiyang Dai, Zicheng Liu, Yumao Lu, Lijuan Wang

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV

2111.07783 2021-11-16 cs.CV cs.LG 70%

FILIP: Fine-grained Interactive Language-Image Pre-Training

Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, Chunjing Xu

专题命中图文多模态：cross-modal(abstract)；image-text(abstract)；分类 cs.CV

2106.13488 2021-11-10 cs.CV 70%

Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training

Hongwei Xue, Yupan Huang, Bei Liu, Houwen Peng, Jianlong Fu, Houqiang Li, Jiebo Luo

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted by NeurIPS 2021

2106.11097 2021-06-22 cs.CV 70%

CLIP2Video: Mastering Video-Text Retrieval via Image CLIP

Han Fang, Pengfei Xiong, Luhui Xu, Yu Chen

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV

2105.14538 2021-06-01 cs.CV cs.AI cs.CL cs.MM 70%

Longer Version for "Deep Context-Encoding Network for Retinal Image Captioning"

Jia-Hong Huang, Ting-Wei Wu, Chao-Han Huck Yang, Marcel Worring

专题命中图文多模态：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments This paper is a longer version of "Deep Context-Encoding Network for Retinal Image Captioning" which is accepted by IEEE International Conference on Image Processing (ICIP), 2021

2104.03135 2021-04-09 cs.CV 70%

Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning

Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, Jianlong Fu

专题命中图文多模态：cross-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted by CVPR2021 oral

2103.07829 2021-03-16 cs.CL 70%

SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels

Chenliang Li, Ming Yan, Haiyang Xu, Fuli Luo, Wei Wang, Bin Bi, Songfang Huang

专题命中图文多模态：cross-modal(abstract)；image-text(abstract)；分类 cs.CL

Comments 10 pages, 4 figures

1908.01665 2019-08-20 cs.CL 70%

Predicting Actions to Help Predict Translations

Zixiu Wu, Julia Ive, Josiah Wang, Pranava Madhyastha, Lucia Specia

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CL

Comments Accepted to workshop "The How2 Challenge: New Tasks for Vision & Language" of International Conference on Machine Learning 2019

1807.11605 2018-08-01 cs.CL 70%

Doubly Attentive Transformer Machine Translation

Hasan Sait Arslan, Mark Fishel, Gholamreza Anbarjafari

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CL

1806.04284 2018-06-13 cs.CL cs.AI cs.CV cs.LG cs.MM 70%

iParaphrasing: Extracting Visually Grounded Paraphrases via an Image

Chenhui Chu, Mayu Otani, Yuta Nakashima

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments COLING 2018

2509.16163 2025-09-22 cs.CV cs.AI cs.CL 69%

Robust Vision-Language Models via Tensor Decomposition: A Defense Against Adversarial Attacks

Het Patel, Muzammil Allie, Qian Zhang, Jia Chen, Evangelos E. Papalexakis

机构 * University of California, Riverside（加州大学河滨分校）

专题命中图文多模态：multimodal(abstract,comments)；分类 cs.CV、cs.CL、cs.AI

Comments To be presented as a poster at the Workshop on Safe and Trustworthy Multimodal AI Systems (SafeMM-AI), 2025

2409.09269 2024-12-13 cs.CV cs.AI cs.CL cs.LG 69%

Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types

Neelabh Sinha, Vinija Jain, Aman Chadha

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI；multi-modal(comments)

Comments Accepted at The First Workshop of Evaluation of Multi-Modal Generation (EvalMG) in 31st International Conference on Computational Linguistics (COLING), 2025. 8 pages + references + 6 pages of Appendix

2010.15251 2021-03-01 cs.CV cs.AI cs.CL cs.LG 69%

Fusion Models for Improved Visual Captioning

Marimuthu Kalimuthu, Aditya Mogadala, Marius Mosbach, Dietrich Klakow

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI；multi-modal(comments)

Comments Accepted at "Multi-Modal Deep Learning: Challenges and Applications" (MMDLCA), International Conference on Pattern Recognition (ICPR)-2020, Milano, Italia

Journal ref Springer LNCS, volume 12666, 2021

2406.09388 2024-06-14 cs.CV cs.AI cs.LG 68%

Exploring the Spectrum of Visio-Linguistic Compositionality and Recognition

Youngtaek Oh, Pyunghwan Ahn, Jinhyung Kim, Gwangmo Song, Soonyoung Lee, In So Kweon, Junmo Kim

专题命中图文多模态：image-text(abstract)；分类 cs.CV、cs.AI；multimodal(comments)；multimodal foundation model(comments)

Comments Accepted to CVPRW 2024 on 'What is Next in Multimodal Foundation Models?'. Code: https://github.com/ytaek-oh/vl_compo

2608.21773 2026-08-25 cs.CR eess.SP 新提交 67%

Privacy Preserving Semantic Communications in Wireless Edge Networks with Vision Language Models

基于视觉语言模型的无线边缘网络中隐私保护语义通信

Haoran Chang, Mingzhe Chen, Qianqian Zhang

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)

AI总结该研究针对无线边缘网络语义通信的隐私泄露问题，提出基于VLM的隐私保护语义通信框架，通过私有区域移除、加密收发器和语义信息瓶颈实现隐私保护，同时保证重构质量并抑制跨设备冗余。

2608.00013 2026-08-25 cs.CL cs.AI cs.CV 版本更新 67%

What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs

从文本到视觉的可迁移性：视觉语言模型（VLM）的能力缩放定律与迁移动态

Ziran Li, Qiang Wang, Zhengyu Chen, Shanglin Lei, Borun Chen, Jingang Wang, Xunliang Cai

机构 * Meituan（美团）； Tsinghua University（清华大学）

专题命中图文多模态：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结该研究提出首个跨家族的能力驱动多模态缩放定律，可通过LLM文本能力预测VLM性能，将主干选择从经验搜索转为定量决策，还揭示了LLM作为VLM主干的相关见解。

2510.22282 2026-08-14 cs.CV cs.AI cs.CL 版本更新 67%

CityRiSE: Reasoning Urban Socio-Economic Status in Large Vision-Language Models via Reinforcement Learning

CityRiSE：基于强化学习的大视觉语言模型城市社会经济地位推理框架

Tianhui Liu, Hetian Pang, Xin Zhang, Jie Feng, Pan Hui, Yong Li

机构 * Information Hub, The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）信息中心）； Department of Electronic Engineering, BNRist, Tsinghua University（清华大学电子工程系）

专题命中图文多模态：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结本研究提出CityRiSE框架，结合强化学习与大视觉语言模型，提升城市社会经济感知的预测准确性与泛化能力，尤其在未见城市和指标上表现优异。

Comments Accepted by ACM MM 2026, https://github.com/tsinghua-fib-lab/CityRiSE

1. 图文多模态 4672 篇

VL-NMS: Breaking Proposal Bottlenecks in Two-Stage Visual-Language Matching

NLIP: Noise-robust Language-Image Pre-training

Efficient Image Captioning for Edge Devices

Prompt-based Learning for Unpaired Image Captioning

Generative Negative Text Replay for Continual Vision-Language Pretraining

CLIP-Driven Fine-grained Text-Image Person Re-identification

Contrastive Language-Image Pre-Training with Knowledge Graphs

Fine-Grained Semantically Aligned Vision-Language Pre-Training

Retrieval-Augmented Transformer for Image Captioning

VLMAE: Vision-Language Masked Autoencoder

Grounded Language-Image Pre-training

Comprehending and Ordering Semantics for Image Captioning

A Better Loss for Visual-Textual Grounding

UFO: A UniFied TransfOrmer for Vision-Language Representation Learning

FILIP: Fine-grained Interactive Language-Image Pre-Training

Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training

CLIP2Video: Mastering Video-Text Retrieval via Image CLIP

Longer Version for "Deep Context-Encoding Network for Retinal Image Captioning"

Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning

SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels

Predicting Actions to Help Predict Translations

Doubly Attentive Transformer Machine Translation

iParaphrasing: Extracting Visually Grounded Paraphrases via an Image

Robust Vision-Language Models via Tensor Decomposition: A Defense Against Adversarial Attacks

Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types

Fusion Models for Improved Visual Captioning

Exploring the Spectrum of Visio-Linguistic Compositionality and Recognition

Privacy Preserving Semantic Communications in Wireless Edge Networks with Vision Language Models

What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs

CityRiSE: Reasoning Urban Socio-Economic Status in Large Vision-Language Models via Reinforcement Learning