多模态大模型 - arXivDaily 专题

2407.19679 2024-07-30 cs.CV cs.AI 73%

Harnessing Large Vision and Language Models in Agriculture: A Review

Hongyan Zhu, Shuai Qin, Min Su, Chengzhi Lin, Anjie Li, Junfeng Gao

专题命中图文多模态：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

2403.17839 2024-07-26 cs.CV cs.AI 73%

ReMamber: Referring Image Segmentation with Mamba Twister

Yuhuan Yang, Chaofan Ma, Jiangchao Yao, Zhun Zhong, Ya Zhang, Yanfeng Wang

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments ECCV 2024

2407.08966 2024-07-15 cs.CV cs.AI cs.LG 73%

LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models

Yabin Zhang, Wenjie Zhu, Chenhang He, Lei Zhang

专题命中图文多模态：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments ECCV2024; Codes and Supp. are available at: https://github.com/YBZh/LAPT

2406.10902 2024-06-18 cs.CV cs.CL 73%

Light Up the Shadows: Enhance Long-Tailed Entity Grounding with Concept-Guided Vision-Language Models

Yikai Zhang, Qianyu He, Xintao Wang, Siyu Yuan, Jiaqing Liang, Yanghua Xiao

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

2405.05524 2024-05-11 cs.CV cs.MM 73%

Universal Adversarial Perturbations for Vision-Language Pre-trained Models

Peng-Fei Zhang, Zi Huang, Guangdong Bai

专题命中图文多模态：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments 9 pages, 5 figures

2404.15946 2024-04-25 cs.CV cs.AI eess.IV 73%

Mammo-CLIP: Leveraging Contrastive Language-Image Pre-training (CLIP) for Enhanced Breast Cancer Diagnosis with Multi-view Mammography

Xuxin Chen, Yuheng Li, Mingzhe Hu, Ella Salari, Xiaoqian Chen, Richard L. J. Qiu, Bin Zheng, Xiaofeng Yang

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

2202.09061 2024-04-19 cs.CV cs.CL 73%

VLP: A Survey on Vision-Language Pre-training

Feilong Chen, Duzhen Zhang, Minglun Han, Xiuyi Chen, Jing Shi, Shuang Xu, Bo Xu

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments A Survey on Vision-Language Pre-training

Journal ref Machine Intelligence Research. 20(1), February 2023, 38-56

2305.07490 2024-04-08 cs.CL cs.CV 73%

ArtGPT-4: Towards Artistic-understanding Large Vision-Language Models with Enhanced Adapter

Zhengqing Yuan, Yunhong He, Kun Wang, Yanfang Ye, Lichao Sun

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

2404.01156 2024-04-02 cs.CV cs.AI 73%

SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining

Chull Hwan Song, Taebaek Hwang, Jooyoung Yoon, Shunghyun Choi, Yeong Hyeon Gu

专题命中图文多模态：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments CVPR2024 Accepted

2403.17995 2024-03-28 cs.CV cs.AI cs.LG 73%

Semi-Supervised Image Captioning Considering Wasserstein Graph Matching

Yang Yang

专题命中图文多模态：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

2312.05588 2024-03-05 cs.AI cs.CV 73%

Language-assisted Vision Model Debugger: A Sample-Free Approach to Finding and Fixing Bugs

Chaoquan Jiang, Jinqiang Wang, Rui Hu, Jitao Sang

专题命中图文多模态：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments 10 pages,8 figures

2312.08846 2024-02-27 cs.LG cs.CL cs.CV 73%

TiMix: Text-aware Image Mixing for Effective Vision-Language Pre-training

Chaoya Jiang, Wei ye, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Shikun Zhang

专题命中图文多模态：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted on AAAI2024

2402.13936 2024-02-22 cs.CL cs.CV 73%

Distinctive Image Captioning: Leveraging Ground Truth Captions in CLIP Guided Reinforcement Learning

Antoine Chaffin, Ewa Kijak, Vincent Claveau

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

2305.15072 2024-02-20 cs.CV cs.MM 73%

PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology

Yuxuan Sun, Chenglu Zhu, Sunyi Zheng, Kai Zhang, Lin Sun, Zhongyi Shui, Yunlong Zhang, Honglin Li, Lin Yang

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.MM

Comments 9 pages, 7 figures

2401.16420 2024-01-30 cs.CV cs.CL 73%

InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model

Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, Jiaqi Wang

专题命中图文多模态：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

Comments Code and models are available at https://github.com/InternLM/InternLM-XComposer

2311.15826 2023-11-28 cs.CV cs.AI 73%

GeoChat: Grounded Large Vision-Language Model for Remote Sensing

Kartik Kuckreja, Muhammad Sohail Danish, Muzammal Naseer, Abhijit Das, Salman Khan, Fahad Shahbaz Khan

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments 10 pages, 4 figures

2210.04135 2023-10-31 cs.CV cs.LG cs.MM 73%

VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment

Shraman Pramanick, Li Jing, Sayan Nag, Jiachen Zhu, Hardik Shah, Yann LeCun, Rama Chellappa

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.MM

Comments Published in TMLR 2023

2303.11403 2023-10-30 cs.CV cs.CL cs.LG 73%

eP-ALM: Efficient Perceptual Augmentation of Language Models

Mustafa Shukor, Corentin Dancette, Matthieu Cord

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Accepted at ICCV 2023. Project page: https://mshukor.github.io/eP-ALM.github.io/

2310.04550 2023-10-10 cs.CV cs.CL cs.LG 73%

Module-wise Adaptive Distillation for Multimodality Foundation Models

Chen Liang, Jiahui Yu, Ming-Hsuan Yang, Matthew Brown, Yin Cui, Tuo Zhao, Boqing Gong, Tianyi Zhou

专题命中图文多模态：multimodal(abstract)；multimodal foundation model(abstract)；分类 cs.CV、cs.CL

2308.12956 2023-08-25 cs.CV cs.AI cs.LG 73%

DLIP: Distilling Language-Image Pre-training

Huafeng Kuang, Jie Wu, Xiawu Zheng, Ming Li, Xuefeng Xiao, Rui Wang, Min Zheng, Rongrong Ji

专题命中图文多模态：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

2308.11194 2023-08-23 cs.CV cs.AI 73%

ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World Data

Maya Varma, Jean-Benoit Delbrouck, Sarah Hooper, Akshay Chaudhari, Curtis Langlotz

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments ICCV 2023

2308.08088 2023-08-17 cs.CV cs.IR cs.MM 73%

Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detection

Rui Cao, Ming Shan Hee, Adriel Kuek, Wen-Haw Chong, Roy Ka-Wei Lee, Jing Jiang

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Camera-ready for 23, ACM MM

2211.12402 2023-08-01 cs.CV cs.CL 73%

X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks

Yan Zeng, Xinsong Zhang, Hang Li, Jiawei Wang, Jipeng Zhang, Wangchunshu Zhou

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments 25 pages

2209.06794 2023-06-06 cs.CV cs.CL 73%

PaLI: A Jointly-Scaled Multilingual Language-Image Model

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, Radu Soricut

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments ICLR 2023 (Notable-top-5%)

2306.00890 2023-06-02 cs.CV cs.CL 73%

LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day

Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, Jianfeng Gao

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments 17 pages; Website: https://aka.ms/llava-med

2302.12971 2023-05-16 cs.CV cs.AI cs.HC 73%

BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding

Yulong Liu, Yongqiang Ma, Wei Zhou, Guibo Zhu, Nanning Zheng

专题命中图文多模态：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

2212.03640 2023-03-28 cs.CV cs.AI 73%

Fine-tuned CLIP Models are Efficient Video Learners

Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, Fahad Shahbaz Khan

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments Accepted at CVPR 2023

2206.07699 2023-02-20 cs.CV cs.CL cs.LG 73%

Write and Paint: Generative Vision-Language Models are Unified Modal Learners

Shizhe Diao, Wangchunshu Zhou, Xinsong Zhang, Jiawei Wang

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments ICLR 2023

2206.07643 2022-11-21 cs.CV cs.CL cs.LG 73%

Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone

Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, Jianfeng Gao, Lijuan Wang

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments NeurIPS 2022. Project Website: https://ashkamath.github.io/FIBER_page

2211.00262 2022-11-02 cs.CL cs.CV 73%

Training Vision-Language Models with Less Bimodal Supervision

Elad Segal, Ben Bogin, Jonathan Berant

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments AKBC 2022

1. 图文多模态 4672 篇

Harnessing Large Vision and Language Models in Agriculture: A Review

ReMamber: Referring Image Segmentation with Mamba Twister

LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models

Light Up the Shadows: Enhance Long-Tailed Entity Grounding with Concept-Guided Vision-Language Models

Universal Adversarial Perturbations for Vision-Language Pre-trained Models

Mammo-CLIP: Leveraging Contrastive Language-Image Pre-training (CLIP) for Enhanced Breast Cancer Diagnosis with Multi-view Mammography

VLP: A Survey on Vision-Language Pre-training

ArtGPT-4: Towards Artistic-understanding Large Vision-Language Models with Enhanced Adapter

SyncMask: Synchronized Attentional Masking for Fashion-centric Vision-Language Pretraining

Semi-Supervised Image Captioning Considering Wasserstein Graph Matching

Language-assisted Vision Model Debugger: A Sample-Free Approach to Finding and Fixing Bugs

TiMix: Text-aware Image Mixing for Effective Vision-Language Pre-training

Distinctive Image Captioning: Leveraging Ground Truth Captions in CLIP Guided Reinforcement Learning

PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology

InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model

GeoChat: Grounded Large Vision-Language Model for Remote Sensing

VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment

eP-ALM: Efficient Perceptual Augmentation of Language Models

Module-wise Adaptive Distillation for Multimodality Foundation Models

DLIP: Distilling Language-Image Pre-training

ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World Data

Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detection

X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks

PaLI: A Jointly-Scaled Multilingual Language-Image Model

LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day

BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding

Fine-tuned CLIP Models are Efficient Video Learners

Write and Paint: Generative Vision-Language Models are Unified Modal Learners

Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone

Training Vision-Language Models with Less Bimodal Supervision