多模态大模型

Evaluating Linguistic Capabilities of Multimodal LLMs in the Lens of Few-Shot Learning

Mustafa Dogan, Ilker Kesen, Iacer Calixto, Aykut Erdem, Erkut Erdem

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Preprint. 33 pages, 17 Figures, 3 Tables

Emergent Visual-Semantic Hierarchies in Image-Text Representations

Morris Alper, Hadar Averbuch-Elor

专题命中 图文多模态 ：image-text(title)；multimodal(abstract)；multimodal foundation model(abstract)；分类 cs.CV、cs.CL

Comments Accepted to ECCV 2024. Project page: https://hierarcaps.github.io/

OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text

Qingyun Li, Zhe Chen, Weiyun Wang, Wenhai Wang, Shenglong Ye, Zhenjiang Jin, Guanzhou Chen, Yinan He, Zhangwei Gao, Erfei Cui, Jiashuo Yu, Hao Tian, Jiasheng Zhou, Chao Xu, Bin Wang, Xingjian Wei, Wei Li, Wenjian Zhang, Bo Zhang, Pinlong Cai, Licheng Wen, Xiangchao Yan, Zhenxiang Li, Pei Chu, Yi Wang, Min Dou, Changyao Tian, Xizhou Zhu, Lewei Lu, Yushi Chen, Junjun He, Zhongying Tu, Tong Lu, Yali Wang, Limin Wang, Dahua Lin, Yu Qiao, Botian Shi, Conghui He, Jifeng Dai

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Improving Adversarial Transferability of Vision-Language Pre-training Models through Collaborative Multimodal Interaction

Jiyuan Fu, Zhaoyu Chen, Kaixun Jiang, Haijing Guo, Jiafeng Wang, Shuyong Gao, Wenqiang Zhang

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.MM

Comments This work won first place in CVPR 2024 Workshop Challenge: Black-box Adversarial Attacks on Vision Foundation Models

TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones

Zhengqing Yuan, Zhaoxu Li, Weiran Huang, Yanfang Ye, Lichao Sun

专题命中 图文多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL

Comments Accepted by ICML workshop 2024

TextBind: Multi-turn Interleaved Multimodal Instruction-following in the Wild

Huayang Li, Siheng Li, Deng Cai, Longyue Wang, Lemao Liu, Taro Watanabe, Yujiu Yang, Shuming Shi

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CL、cs.AI

Comments Findings of ACL 2024

LLMs as Bridges: Reformulating Grounded Multimodal Named Entity Recognition

Jinyuan Li, Han Li, Di Sun, Jiahao Wang, Wenkun Zhang, Zan Wang, Gang Pan

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Accepted to Findings of ACL 2024

Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification

Chao Yi, Lu Ren, De-Chuan Zhan, Han-Jia Ye

专题命中 图文多模态 ：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

UrbanCross: Enhancing Satellite Image-Text Retrieval with Cross-Domain Adaptation

Siru Zhong, Xixuan Hao, Yibo Yan, Ying Zhang, Yangqiu Song, Yuxuan Liang

专题命中 图文多模态 ：image-text(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, Yinfei Yang

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Troika: Multi-Path Cross-Modal Traction for Compositional Zero-Shot Learning

Siteng Huang, Biao Gong, Yutong Feng, Min Zhang, Yiliang Lv, Donglin Wang

专题命中 图文多模态 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

Comments CVPR 2024

Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities

Yiyuan Zhang, Xiaohan Ding, Kaixiong Gong, Yixiao Ge, Ying Shan, Xiangyu Yue

专题命中 图文多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments CVPR 2024. Code and models are available at https://github.com/AILab-CVC/M2PT

DreamLLM: Synergistic Multimodal Comprehension and Creation

Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, Li Yi

专题命中 图文多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL

Comments ICLR 2024 (Spotlight)

RED-DOT: Multimodal Fact-checking via Relevant Evidence Detection

Stefanos-Iordanis Papadopoulos, Christos Koutlis, Symeon Papadopoulos, Panagiotis C. Petrantonakis

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.MM

Improving fine-grained understanding in image-text pre-training

Ioana Bica, Anastasija Ilić, Matthias Bauer, Goker Erdogan, Matko Bošnjak, Christos Kaplanis, Alexey A. Gritsenko, Matthias Minderer, Charles Blundell, Razvan Pascanu, Jovana Mitrović

专题命中 图文多模态 ：image-text(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments 26 pages

Training Transitive and Commutative Multimodal Transformers with LoReTTa

Manuel Tran, Yashin Dicente Cid, Amal Lahiani, Fabian J. Theis, Tingying Peng, Eldad Klaiman

专题命中 图文多模态 ：multimodal(title,abstract)；multimodal foundation model(abstract)；分类 cs.CV、cs.AI

Comments Accepted at NeurIPS 2023 (poster). Camera-ready version

Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models

Xin He, Longhui Wei, Lingxi Xie, Qi Tian

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.MM

CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcare

Akash Ghosh, Arkadeep Acharya, Raghav Jain, Sriparna Saha, Aman Chadha, Setu Sinha

专题命中 图文多模态 ：multimodal(title,abstract)；multimodal foundation model(abstract)；分类 cs.CL、cs.AI

Comments AAAI 2024

SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger

Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu Enwei Zhang, Wei Liu, Jie Yang, Ke Li, Xing Sun

专题命中 图文多模态 ：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Improving Cross-modal Alignment with Synthetic Pairs for Text-only Image Captioning

Zhiyue Liu, Jinyuan Liu, Fanrong Ma

专题命中 图文多模态 ：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments AAAI2024

Delving into Multimodal Prompting for Fine-grained Visual Classification

Xin Jiang, Hao Tang, Junyao Gao, Xiaoyu Du, Shengfeng He, Zechao Li

专题命中 图文多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments This paper is accepted by AAAI 2024

ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Training

Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodolà, Francesco Locatello

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments 17 pages

COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs

Tiep Le, Vasudev Lal, Phillip Howard

专题命中 图文多模态 ：image-text(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL

Comments Accepted to NeurIPS 2023 Datasets and Benchmarks Track

A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding

Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan A. Plummer, Kate Saenko, Jianmo Ni, Mandy Guo

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Accepted in EMNLP 2023, revision contains camera ready edits. Data can be downloaded at https://github.com/google-research-datasets/wit/blob/main/wikiweb2m.md

Aligning Large Multimodal Models with Factually Augmented RLHF

Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, Trevor Darrell

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Preprint

MaMMUT: A Simple Architecture for Joint Learning for MultiModal Tasks

Weicheng Kuo, AJ Piergiovanni, Dahun Kim, Xiyang Luo, Ben Caine, Wei Li, Abhijit Ogale, Luowei Zhou, Andrew Dai, Zhifeng Chen, Claire Cui, Anelia Angelova

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Published in Transactions on Machine Learning Research ( https://jmlr.org/tmlr/ ). 18 pages, 4 figures

Med-Flamingo: a Multimodal Medical Few-shot Learner

Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Cyril Zakka, Yash Dalmia, Eduardo Pontes Reis, Pranav Rajpurkar, Jure Leskovec

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments Preprint

Target-oriented Sentiment Classification with Sequential Cross-modal Semantic Graph

Yufeng Huang, Zhuo Chen, Jiaoyan Chen, Jeff Z. Pan, Zhen Yao, Wen Zhang

专题命中 图文多模态 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Comments ICANN 2023, https://github.com/zjukg/SeqCSG

mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections

Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, Luo Si

专题命中 图文多模态 ：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Journal ref EMNLP2022

Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment

Rohan Pandey, Rulin Shao, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency

专题命中 图文多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL

Comments ACL 2023