多模态大模型 - arXivDaily 专题

2205.12029 2023-05-12 cs.CV 83%

VLCDoC: Vision-Language Contrastive Pre-Training Model for Cross-Modal Document Classification

Souhail Bakkali, Zuheng Ming, Mickael Coustaty, Marçal Rusiñol, Oriol Ramos Terrades

专题命中图文多模态：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments Accepted at PR

2305.04530 2023-05-09 cs.CL 83%

A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues

Yunxin Li, Baotian Hu, Xinyu Chen, Yuxin Ding, Lin Ma, Min Zhang

专题命中图文多模态：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CL

Comments Accepted to ACL 2023 Main Conference

2304.03307 2023-04-10 cs.CV eess.IV 83%

Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting

Syed Talal Wasim, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, Mubarak Shah

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted at CVPR-2023. Codes/models available at https://github.com/TalalWasim/Vita-CLIP

2210.03117 2023-04-04 cs.CV 83%

MaPLe: Multi-modal Prompt Learning

Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, Fahad Shahbaz Khan

专题命中图文多模态：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments Accepted at CVPR2023

2301.00182 2023-03-28 cs.CV 83%

Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models

Wenhao Wu, Xiaohan Wang, Haipeng Luo, Jingdong Wang, Yi Yang, Wanli Ouyang

专题命中图文多模态：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted by CVPR 2023

2303.06591 2023-03-14 cs.CV 83%

Accommodating Audio Modality in CLIP for Multimodal Processing

Ludan Ruan, Anwen Hu, Yuqing Song, Liang Zhang, Sipeng Zheng, Qin Jin

专题命中图文多模态：multimodal(title,abstract)；vision-language-audio(abstract)；分类 cs.CV

Comments Accepted by AAAI2023

2209.06430 2023-03-03 cs.CV 83%

CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment

Hongwei Xue, Yuchong Sun, Bei Liu, Jianlong Fu, Ruihua Song, Houqiang Li, Jiebo Luo

专题命中图文多模态：image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by ICLR 2023

2207.01887 2023-02-02 cs.CV 83%

Open-Vocabulary Multi-Label Classification via Multi-Modal Knowledge Transfer

Sunan He, Taian Guo, Tao Dai, Ruizhi Qiao, Bo Ren, Shu-Tao Xia

专题命中图文多模态：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments AAAI 2023 (Oral presentation paper). Updated version

2106.12570 2022-12-19 cs.LG cs.CV 83%

Learning Multimodal VAEs through Mutual Supervision

Tom Joy, Yuge Shi, Philip H. S. Torr, Tom Rainforth, Sebastian M. Schmon, N. Siddharth

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

2203.14713 2022-11-21 cs.IR cs.AI 83%

Image-text Retrieval: A Survey on Recent Research and Development

Min Cao, Shiping Li, Juntao Li, Liqiang Nie, Min Zhang

专题命中图文多模态：image-text(title,abstract)；cross-modal(abstract)；分类 cs.AI

Comments Accpted by IJCAI'2022 survey track

2210.13188 2022-10-25 cs.CV 83%

Dissecting Deep Metric Learning Losses for Image-Text Retrieval

Hong Xuan, Xi Chen

专题命中图文多模态：image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments arXiv admin note: text overlap with arXiv:2201.11307

Journal ref WACV2023

2205.14204 2022-10-24 cs.CV 83%

Multimodal Masked Autoencoders Learn Transferable Representations

Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, Pieter Abbeel

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

2111.11430 2022-07-20 cs.CV 83%

Class-agnostic Object Detection with Multi-modal Transformer

Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, Ming-Hsuan Yang

专题命中图文多模态：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted at ECCV 2022

2201.06686 2022-06-07 cs.CV 83%

Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching

Hengcan Shi, Munawar Hayat, Jianfei Cai

专题命中图文多模态：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments 9 pages, 7 figures

2204.04014 2022-05-06 cs.CV cs.NE 83%

Multimodal Quasi-AutoRegression: Forecasting the visual popularity of new fashion products

Stefanos I. Papadopoulos, Christos Koutlis, Symeon Papadopoulos, Ioannis Kompatsiaris

专题命中图文多模态：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments 14 pages, 4 figures

2204.05533 2022-04-28 cs.CL cs.SI 83%

How does fake news use a thumbnail? CLIP-based Multimodal Detection on the Unrepresentative News Image

Hyewon Choi, Yejun Yoon, Seunghyun Yoon, Kunwoo Park

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CL

Comments 9 pages, 8 figures including appendix figure, 2 tables. Published in Findings of ACL workshop, CONSTRAINT 2022 (Long paper). The manuscript is slightly revised after the camera ready version

2204.04588 2022-04-12 cs.CV cs.LG 83%

Robust Cross-Modal Representation Learning with Progressive Self-Distillation

Alex Andonian, Shixing Chen, Raffay Hamid

专题命中图文多模态：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted to CVPR 2022

2202.06076 2022-02-15 cs.CV cs.LG 83%

Indication as Prior Knowledge for Multimodal Disease Classification in Chest Radiographs with Transformers

Grzegorz Jacenków, Alison Q. O'Neil, Sotirios A. Tsaftaris

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted at the IEEE International Symposium on Biomedical Imaging (ISBI) 2022 as an oral presentation

2108.08217 2021-08-19 cs.CV cs.AI cs.CL cs.LG cs.MM 83%

X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics

Yehao Li, Yingwei Pan, Jingwen Chen, Ting Yao, Tao Mei

专题命中图文多模态：cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by 2021 ACMMM Open Source Software Competition. Source code: https://github.com/YehLi/xmodaler

2107.14596 2021-08-02 cs.CL 83%

Multi-stage Pre-training over Simplified Multimodal Pre-training Models

Tongtong Liu, Fangxiang Feng, Xiaojie Wang

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CL

Comments 10 pages, 4 figures

2107.11970 2021-07-27 cs.CV 83%

Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph

Wentian Zhao, Yao Hu, Heda Wang, Xinxiao Wu, Jiebo Luo

专题命中图文多模态：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2107.10300 2021-07-23 cs.CV cs.AI cs.CL cs.LG cs.MM eess.IV 83%

iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretability

Aman Chadha, Vinija Jain

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 12 pages, 1 figure, 7 tables

2106.06509 2021-06-14 cs.CV 83%

Step-Wise Hierarchical Alignment Network for Image-Text Matching

Zhong Ji, Kexin Chen, Haoran Wang

专题命中图文多模态：image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by IJCAI 2021

2104.12471 2021-04-27 cs.CV cs.AI cs.CL cs.IR cs.MM 83%

Contextualized Keyword Representations for Multi-modal Retinal Image Captioning

Jia-Hong Huang, Ting-Wei Wu, Marcel Worring

专题命中图文多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments This paper is accepted by ACM International Conference on Multimedia Retrieval (ICMR), 2021

2104.10406 2021-04-22 cs.CV 83%

Discrete-continuous Action Space Policy Gradient-based Attention for Image-Text Matching

Shiyang Yan, Li Yu, Yuan Xie

专题命中图文多模态：image-text(title,abstract)；multi-modal(abstract)；分类 cs.CV

2006.09310 2020-06-17 cs.CV cs.LG 83%

Deep Multimodal Transfer-Learned Regression in Data-Poor Domains

Levi McClenny, Mulugeta Haile, Vahid Attari, Brian Sadler, Ulisses Braga-Neto, Raymundo Arroyave

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

1908.06066 2019-12-04 cs.CV 83%

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, Ming Zhou

专题命中图文多模态：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments accepted by AAAI-2020. arXiv admin note: text overlap with arXiv:1909.11740 by other authors

1907.12490 2019-07-30 cs.IR cs.MM 83%

Deep Cross-Modal Hashing with Hashing Functions and Unified Hash Codes Jointly Learning

Rong-Cheng Tu, Xian-Ling Mao, Bing Ma, Yong Hu, Tan Yan, Wei Wei, Heyan Huang

专题命中图文多模态：cross-modal(title,abstract)；image-text(abstract)；分类 cs.MM

1906.09610 2019-06-25 cs.CV 83%

Improving Description-based Person Re-identification by Multi-granularity Image-text Alignments

Kai Niu, Yan Huang, Wanli Ouyang, Liang Wang

专题命中图文多模态：image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV

1904.06286 2019-04-17 cs.CL 83%

CITE: A Corpus of Image-Text Discourse Relations

Malihe Alikhani, Sreyasi Nag Chowdhury, Gerard de Melo, Matthew Stone

专题命中图文多模态：image-text(title,abstract)；multimodal(abstract)；分类 cs.CL

Comments 7 pages

Journal ref 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics

1. 图文多模态 4682 篇

VLCDoC: Vision-Language Contrastive Pre-Training Model for Cross-Modal Document Classification

A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues

Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting

MaPLe: Multi-modal Prompt Learning

Bidirectional Cross-Modal Knowledge Exploration for Video Recognition with Pre-trained Vision-Language Models

Accommodating Audio Modality in CLIP for Multimodal Processing

CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment

Open-Vocabulary Multi-Label Classification via Multi-Modal Knowledge Transfer

Learning Multimodal VAEs through Mutual Supervision

Image-text Retrieval: A Survey on Recent Research and Development

Dissecting Deep Metric Learning Losses for Image-Text Retrieval

Multimodal Masked Autoencoders Learn Transferable Representations

Class-agnostic Object Detection with Multi-modal Transformer

Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching

Multimodal Quasi-AutoRegression: Forecasting the visual popularity of new fashion products

How does fake news use a thumbnail? CLIP-based Multimodal Detection on the Unrepresentative News Image

Robust Cross-Modal Representation Learning with Progressive Self-Distillation

Indication as Prior Knowledge for Multimodal Disease Classification in Chest Radiographs with Transformers

X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics

Multi-stage Pre-training over Simplified Multimodal Pre-training Models

Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph

iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretability

Step-Wise Hierarchical Alignment Network for Image-Text Matching

Contextualized Keyword Representations for Multi-modal Retinal Image Captioning

Discrete-continuous Action Space Policy Gradient-based Attention for Image-Text Matching

Deep Multimodal Transfer-Learned Regression in Data-Poor Domains

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

Deep Cross-Modal Hashing with Hashing Functions and Unified Hash Codes Jointly Learning

Improving Description-based Person Re-identification by Multi-granularity Image-text Alignments

CITE: A Corpus of Image-Text Discourse Relations