多模态大模型

CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance

Chu Myaet Thwal, Ye Lin Tun, Minh N. H. Nguyen, Eui-Nam Huh, Choong Seon Hong

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 14 pages, 5 figures, 24 tables

RetinalGPT: A Retinal Clinical Preference Conversational Assistant Powered by Large Vision-Language Models

Wenhui Zhu, Xin Li, Xiwen Chen, Peijie Qiu, Vamsi Krishna Vasa, Xuanzhao Dong, Yanxi Chen, Natasha Lepore, Oana Dumitrascu, Yi Su, Yalin Wang

专题命中 图文多模态 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

FaultGPT: Industrial Fault Diagnosis Question Answering System by Vision Language Models

Jiao Chen, Ruyi Huang, Zuohong Lv, Jianhua Tang, Weihua Li

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)

Unleashing Text-to-Image Diffusion Prior for Zero-Shot Image Captioning

Jianjie Luo, Jingwen Chen, Yehao Li, Yingwei Pan, Jianlin Feng, Hongyang Chao, Ting Yao

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments ECCV 2024

Uni-Mlip: Unified Self-supervision for Medical Vision Language Pre-training

Ameera Bawazir, Kebin Wu, Wenbin Li

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 15 pages, 2 figures, accepted by BMVC'24

Identifying Implicit Social Biases in Vision-Language Models

Kimia Hamidieh, Haoran Zhang, Walter Gerych, Thomas Hartvigsen, Marzyeh Ghassemi

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

HSVLT: Hierarchical Scale-Aware Vision-Language Transformer for Multi-Label Image Classification

Shuyi Ouyang, Hongyi Wang, Ziwei Niu, Zhenjia Bai, Shiao Xie, Yingying Xu, Ruofeng Tong, Yen-Wei Chen, Lanfen Lin

专题命中 图文多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 10 pages, 6 figures

Journal ref Proceedings of the 31st ACM International Conference on Multimedia. 2023: 4768-4777

Advancing High Resolution Vision-Language Models in Biomedicine

Zekai Chen, Arda Pekis, Kevin Brown

专题命中 图文多模态 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 15 pages

Improved Baselines with Visual Instruction Tuning

Haotian Liu, Chunyuan Li, Yuheng Li, Yong Jae Lee

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Camera ready, CVPR 2024 (highlight). LLaVA project page: https://llava-vl.github.io

Frozen Transformers in Language Models Are Effective Visual Encoder Layers

Ziqi Pang, Ziyang Xie, Yunze Man, Yu-Xiong Wang

专题命中 图文多模态 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ICLR 2024 Spotlight. 23 pages, 13 figures. Code at https://github.com/ziqipang/LM4VisualEncoding

LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation

Suhyeon Lee, Won Jun Kim, Jinho Chang, Jong Chul Ye

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 21 pages, 8 figures; ICLR 2024 (poster)

EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE

Junyi Chen, Longteng Guo, Jia Sun, Shuai Shao, Zehuan Yuan, Liang Lin, Dongyu Zhang

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted by AAAI 2024

Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias

Zhongwei Wan, Che Liu, Mi Zhang, Jie Fu, Benyou Wang, Sibo Cheng, Lei Ma, César Quilodrán-Casas, Rossella Arcucci

专题命中 图文多模态 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NeurIPS 2023 Main track

Analyzing Zero-Shot Abilities of Vision-Language Models on Video Understanding Tasks

Avinash Madasu, Anahita Bhiwandiwalla, Vasudev Lal

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare

Junling Liu, Ziming Wang, Qichen Ye, Dading Chong, Peilin Zhou, Yining Hua

专题命中 图文多模态 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation

Hongguang Zhu, Yunchao Wei, Xiaodan Liang, Chunjie Zhang, Yao Zhao

专题命中 图文多模态 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by ICCV 2023. Code: https://github.com/KevinLight831/CTP

Enhancing Dynamic Image Advertising with Vision-Language Pre-training

Zhoufutu Wen, Xinyu Zhao, Zhipeng Jin, Yi Yang, Wei Jia, Xiaodong Chen, Shuanglong Li, Lin Liu

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)

Comments 6 pages, 3 figures, accepted to SIRIP 2023

Fine-grained Image Captioning with CLIP Reward

Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, Mohit Bansal

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NAACL Findings 2022

MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting

Oscar Mañas, Pau Rodriguez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, Aishwarya Agrawal

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at EACL 2023 (main track); 26 pages, 21 figures, 6 tables; Pau Rodriguez and Saba Ahmadi had equal contributions

ViewCo: Discovering Text-Supervised Segmentation Masks via Multi-View Semantic Consistency

Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guangrun Wang, Jianzhuang Liu, Xiaojun Chang, Xiaodan Liang

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention

Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, Bin Cui

专题命中 图文多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by AAAI 2023, 12 pages, 6 figures

MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning

Zejun Li, Zhihao Fan, Huaixiao Tou, Jingjing Chen, Zhongyu Wei, Xuanjing Huang

专题命中 图文多模态 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by ACM MM22

VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers

Estelle Aflalo, Meng Du, Shao-Yen Tseng, Yongfei Liu, Chenfei Wu, Nan Duan, Vasudev Lal

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Best Demo Award at CVPR 2022

Declaration-based Prompt Tuning for Visual Question Answering

Yuhang Liu, Wei Wei, Daowan Peng, Feida Zhu

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted to IJCAI2022, data and codes are available at https://github.com/CCIIPLab/DPT

Vision-Language Pre-Training for Boosting Scene Text Detectors

Sibo Song, Jianqiang Wan, Zhibo Yang, Jun Tang, Wenqing Cheng, Xiang Bai, Cong Yao

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by CVPR 2022

Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?

Sedigheh Eslami, Gerard de Melo, Christoph Meinel

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning

Haiyang Xu, Ming Yan, Chenliang Li, Bin Bi, Songfang Huang, Wenming Xiao, Fei Huang

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL2021 main conference

SBAT: Video Captioning with Sparse Boundary-Aware Transformer

Tao Jin, Siyu Huang, Ming Chen, Yingming Li, Zhongfei Zhang

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Appearing at IJCAI 2020

Mull-Tokens: Modality-Agnostic Latent Thinking

Mull-Tokens: 通用模态的潜在思考

Arijit Ray, Ahmed Abdelkader, Chengzhi Mao, Bryan A. Plummer, Kate Saenko, Ranjay Krishna, Leonidas Guibas, Wen-Sheng Chu

机构 * Google（谷歌） ； University of Washington（华盛顿大学） ； Stanford University（斯坦福大学） ； Boston University（波士顿大学） 纠错

专题命中 图文多模态 ：multimodal(abstract,comments)；image-text(abstract)；分类 cs.CV、cs.AI

AI总结 本文提出Mull-Tokens，一种无需模态切换的潜在令牌，用于空间、时间等多模态推理，通过预训练和微调在四个挑战性基准上实现3%-16%的提升。

Comments Project webpage: https://arijitray.com/multimodal_thinking/, Accepted to CVPR 2026 (Findings Track)

SIA: Enhancing Safety via Intent Awareness for Vision-Language Models

Youngjin Na, Sangheon Jeong, Youngwan Lee, Jian Lee, Dawoon Jeong, Youngman Kim

机构 * VLM Safety LAB, MODULABS（视觉语言模型安全实验室，MODULABS） ； ETRI（电子技术研究院） ； KAIST（韩国科学技术院） 纠错

专题命中 图文多模态 ：multimodal(abstract,comments)；image-text(abstract)；分类 cs.CV、cs.AI

Comments Accepted to Safe and Trustworthy Multimodal AI Systems(SafeMM-AI) Workshop at ICCV2025, Non-archival track