多模态大模型

Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text

Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, Yejin Choi

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments NeurIPS D&B 2023. Project homepage: https://github.com/allenai/mmc4

Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionality

Harman Singh, Pengchuan Zhang, Qifan Wang, Mengjiao Wang, Wenhan Xiong, Jingfei Du, Yu Chen

专题命中 图文多模态 ：image-text(title)；multimodal(abstract)；分类 cs.CV、cs.CL

Comments EMNLP 2023 (long paper, main conference)

Exploring Affordance and Situated Meaning in Image Captions: A Multimodal Analysis

Pin-Er Chen, Po-Ya Angela Wang, Hsin-Yu Chou, Yu-Hsiang Tseng, Shu-Kai Hsieh

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments 10 pages, 9 figures

Leveraging Image-Text Similarity and Caption Modification for the DataComp Challenge: Filtering Track and BYOD Track

Shuhei Yokoo, Peifei Zhu, Yuchi Ishikawa, Mikihiro Tanaka, Masayoshi Kondo, Hirokatsu Kataoka

专题命中 图文多模态 ：image-text(title)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments Accepted at the ICCV 2023 Workshop on Towards the Next Generation of Computer Vision Datasets: DataComp Track

Knowledge Extraction and Distillation from Large-Scale Image-Text Colonoscopy Records Leveraging Large Language and Vision Models

Shuo Wang, Yan Zhu, Xiaoyuan Luo, Zhiwei Yang, Yizhe Zhang, Peiyao Fu, Manning Wang, Zhijian Song, Quanlin Li, Pinghong Zhou, Yike Guo

专题命中 图文多模态 ：image-text(title,abstract)；分类 cs.CV、cs.AI

Multimodal Analysis Of Google Bard And GPT-Vision: Experiments In Visual Reasoning

David Noever, Samantha Elizabeth Miller Noever

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants

Tianyu Yu, Jinyi Hu, Yuan Yao, Haoye Zhang, Yue Zhao, Chongyi Wang, Shan Wang, Yinxv Pan, Jiao Xue, Dahai Li, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Multimodal Neurons in Pretrained Text-Only Transformers

Sarah Schwettmann, Neil Chowdhury, Samuel Klein, David Bau, Antonio Torralba

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Oral presentation at ICCV CLVL 2023

Noise-aware Learning from Web-crawled Image-Text Data for Image Captioning

Wooyoung Kang, Jonghwan Mun, Sungjun Lee, Byungseok Roh

专题命中 图文多模态 ：image-text(title,abstract)；分类 cs.CV、cs.AI

Findings of Factify 2: Multimodal Fake News Detection

S Suryavardan, Shreyash Mishra, Megha Chakraborty, Parth Patwa, Anku Rani, Aman Chadha, Aishwarya Reganti, Amitava Das, Amit Sheth, Manoj Chinnakotla, Asif Ekbal, Srijan Kumar

专题命中 图文多模态 ：multimodal(title)；multi-modal(abstract)；分类 cs.CV、cs.CL

Comments Defactify2 @AAAI 2023

FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning

Haokun Chen, Yao Zhang, Denis Krompass, Jindong Gu, Volker Tresp

专题命中 图文多模态 ：multi-modal(title,abstract)；分类 cs.AI、cs.MM

Multi-modal Pre-training for Medical Vision-language Understanding and Generation: An Empirical Study with A New Benchmark

Li Xu, Bo Liu, Ameer Hamza Khan, Lu Fan, Xiao-Ming Wu

专题命中 图文多模态 ：multi-modal(title)；image-text(abstract)；分类 cs.CV、cs.AI

Comments Published as oral paper in CHIL 2023

CLIP Multi-modal Hashing: A new baseline CLIPMH

Jian Zhu, Mingkai Sheng, Mingda Ke, Zhangmin Huang, Jingfei Chang

专题命中 图文多模态 ：multi-modal(title,abstract)；分类 cs.CV、cs.MM

Comments submit to ICASSP2024

Evaluating Picture Description Speech for Dementia Detection using Image-text Alignment

Youxiang Zhu, Nana Lin, Xiaohui Liang, John A. Batsis, Robert M. Roth, Brian MacWhinney

专题命中 图文多模态 ：image-text(title,abstract)；分类 cs.CV、cs.CL

Your Negative May not Be True Negative: Boosting Image-Text Matching with False Negative Elimination

Haoxuan Li, Yi Bin, Junrong Liao, Yang Yang, Heng Tao Shen

专题命中 图文多模态 ：image-text(title,abstract)；分类 cs.CV、cs.MM

Comments Accepted at ACM MM 2023

Multimodal Color Recommendation in Vector Graphic Documents

Qianru Qiu, Xueting Wang, Mayu Otani

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.MM

Comments Accepted to ACM MM 2023

Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment

Shengqiong Wu, Hao Fei, Wei Ji, Tat-Seng Chua

专题命中 图文多模态 ：cross-modal(title,abstract)；分类 cs.CV、cs.CL

From Association to Generation: Text-only Captioning by Unsupervised Cross-modal Mapping

Junyang Wang, Ming Yan, Yi Zhang, Jitao Sang

专题命中 图文多模态 ：cross-modal(title)；image-text(abstract)；分类 cs.CV、cs.CL

Comments 11 pages, 15 figures, has been accepted by IJCAI 2023

Multi-modal reward for visual relationships-based image captioning

Ali Abedi, Hossein Karshenas, Peyman Adibi

专题命中 图文多模态 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments 18 pages, 10 figures

INO at Factify 2: Structure Coherence based Multi-Modal Fact Verification

Yinuo Zhang, Zhulin Tao, Xi Wang, Tongyue Wang

专题命中 图文多模态 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Comments AAAI-23 DeFactify 2 Workshop (2nd Prize)

Selectively Hard Negative Mining for Alleviating Gradient Vanishing in Image-Text Matching

Zheng Li, Caili Guo, Xin Wang, Zerun Feng, Zhongtian Du

专题命中 图文多模态 ：image-text(title,abstract)；分类 cs.CV、cs.MM

Video Captioning with Guidance of Multimodal Latent Topics

Shizhe Chen, Jia Chen, Qin Jin, Alexander Hauptmann

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments ACM Multimedia 2017

MAViC: Multimodal Active Learning for Video Captioning

Gyanendra Das, Xavier Thomas, Anant Raj, Vikram Gupta

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Cross-Modal Similarity-Based Curriculum Learning for Image Captioning

Hongkuan Zhang, Saku Sugawara, Akiko Aizawa, Lei Zhou, Ryohei Sasano, Koichi Takeda

专题命中 图文多模态 ：cross-modal(title,abstract)；分类 cs.CV、cs.CL

Comments EMNLP 2022

DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention

Fenglin Liu, Xian Wu, Shen Ge, Xuancheng Ren, Wei Fan, Xu Sun, Yuexian Zou

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Published in ACM TKDD2022 (ACM Transactions on Knowledge Discovery from Data)

On the Use of Modality-Specific Large-Scale Pre-Trained Encoders for Multimodal Sentiment Analysis

Atsushi Ando, Ryo Masumura, Akihiko Takashima, Satoshi Suzuki, Naoki Makishima, Keita Suzuki, Takafumi Moriya, Takanori Ashihara, Hiroshi Sato

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted to SLT 2022

Visual Answer Localization with Cross-modal Mutual Knowledge Transfer

Yixuan Weng, Bin Li

专题命中 图文多模态 ：cross-modal(title,abstract)；分类 cs.CV、cs.AI

Comments 4 pages, 3 figures, 2 tables

Bloom Library: Multimodal Datasets in 300+ Languages for a Variety of Downstream Tasks

Colin Leong, Joshua Nemecek, Jacob Mansdorfer, Anna Filighera, Abraham Owodunni, Daniel Whitenack

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 14 pages, 1 figure, 3 tables, accepted to and presented at EMNLP 2022

Journal ref EMNLP 2022

Cross-Modal Alignment Learning of Vision-Language Conceptual Systems

Taehyeong Kim, Hyeonseop Song, Byoung-Tak Zhang

专题命中 图文多模态 ：cross-modal(title,abstract)；分类 cs.CV、cs.AI

Comments 19 pages, 4 figures

Towards Multimodal Vision-Language Models Generating Non-Generic Text

Wes Robbins, Zanyar Zohourianshahzadi, Jugal Kalita

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Journal ref 2021 International Conference on Natural Language Processing