多模态大模型

HSVLT: Hierarchical Scale-Aware Vision-Language Transformer for Multi-Label Image Classification

Shuyi Ouyang, Hongyi Wang, Ziwei Niu, Zhenjia Bai, Shiao Xie, Yingying Xu, Ruofeng Tong, Yen-Wei Chen, Lanfen Lin

专题命中 图文多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 10 pages, 6 figures

Journal ref Proceedings of the 31st ACM International Conference on Multimedia. 2023: 4768-4777

Advancing High Resolution Vision-Language Models in Biomedicine

Zekai Chen, Arda Pekis, Kevin Brown

专题命中 图文多模态 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 15 pages

Improved Baselines with Visual Instruction Tuning

Haotian Liu, Chunyuan Li, Yuheng Li, Yong Jae Lee

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Camera ready, CVPR 2024 (highlight). LLaVA project page: https://llava-vl.github.io

Frozen Transformers in Language Models Are Effective Visual Encoder Layers

Ziqi Pang, Ziyang Xie, Yunze Man, Yu-Xiong Wang

专题命中 图文多模态 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ICLR 2024 Spotlight. 23 pages, 13 figures. Code at https://github.com/ziqipang/LM4VisualEncoding

LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation

Suhyeon Lee, Won Jun Kim, Jinho Chang, Jong Chul Ye

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 21 pages, 8 figures; ICLR 2024 (poster)

EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE

Junyi Chen, Longteng Guo, Jia Sun, Shuai Shao, Zehuan Yuan, Liang Lin, Dongyu Zhang

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted by AAAI 2024

Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Bias

Zhongwei Wan, Che Liu, Mi Zhang, Jie Fu, Benyou Wang, Sibo Cheng, Lei Ma, César Quilodrán-Casas, Rossella Arcucci

专题命中 图文多模态 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NeurIPS 2023 Main track

Analyzing Zero-Shot Abilities of Vision-Language Models on Video Understanding Tasks

Avinash Madasu, Anahita Bhiwandiwalla, Vasudev Lal

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Qilin-Med-VL: Towards Chinese Large Vision-Language Model for General Healthcare

Junling Liu, Ziming Wang, Qichen Ye, Dading Chong, Peilin Zhou, Yining Hua

专题命中 图文多模态 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation

Hongguang Zhu, Yunchao Wei, Xiaodan Liang, Chunjie Zhang, Yao Zhao

专题命中 图文多模态 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by ICCV 2023. Code: https://github.com/KevinLight831/CTP

Enhancing Dynamic Image Advertising with Vision-Language Pre-training

Zhoufutu Wen, Xinyu Zhao, Zhipeng Jin, Yi Yang, Wei Jia, Xiaodong Chen, Shuanglong Li, Lin Liu

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)

Comments 6 pages, 3 figures, accepted to SIRIP 2023

Fine-grained Image Captioning with CLIP Reward

Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, Mohit Bansal

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NAACL Findings 2022

MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Prompting

Oscar Mañas, Pau Rodriguez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, Aishwarya Agrawal

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at EACL 2023 (main track); 26 pages, 21 figures, 6 tables; Pau Rodriguez and Saba Ahmadi had equal contributions

ViewCo: Discovering Text-Supervised Segmentation Masks via Multi-View Semantic Consistency

Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guangrun Wang, Jianzhuang Liu, Xiaojun Chang, Xiaodan Liang

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention

Ziyu Guo, Renrui Zhang, Longtian Qiu, Xianzheng Ma, Xupeng Miao, Xuming He, Bin Cui

专题命中 图文多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by AAAI 2023, 12 pages, 6 figures

MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning

Zejun Li, Zhihao Fan, Huaixiao Tou, Jingjing Chen, Zhongyu Wei, Xuanjing Huang

专题命中 图文多模态 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by ACM MM22

VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers

Estelle Aflalo, Meng Du, Shao-Yen Tseng, Yongfei Liu, Chenfei Wu, Nan Duan, Vasudev Lal

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Best Demo Award at CVPR 2022

Declaration-based Prompt Tuning for Visual Question Answering

Yuhang Liu, Wei Wei, Daowan Peng, Feida Zhu

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted to IJCAI2022, data and codes are available at https://github.com/CCIIPLab/DPT

Vision-Language Pre-Training for Boosting Scene Text Detectors

Sibo Song, Jianqiang Wan, Zhibo Yang, Jun Tang, Wenqing Cheng, Xiang Bai, Cong Yao

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by CVPR 2022

Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?

Sedigheh Eslami, Gerard de Melo, Christoph Meinel

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning

Haiyang Xu, Ming Yan, Chenliang Li, Bin Bi, Songfang Huang, Wenming Xiao, Fei Huang

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL2021 main conference

SBAT: Video Captioning with Sparse Boundary-Aware Transformer

Tao Jin, Siyu Huang, Ming Chen, Yingming Li, Zhongfei Zhang

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Appearing at IJCAI 2020

Mull-Tokens: Modality-Agnostic Latent Thinking

Mull-Tokens: 通用模态的潜在思考

Arijit Ray, Ahmed Abdelkader, Chengzhi Mao, Bryan A. Plummer, Kate Saenko, Ranjay Krishna, Leonidas Guibas, Wen-Sheng Chu

机构 * Google（谷歌） ； University of Washington（华盛顿大学） ； Stanford University（斯坦福大学） ； Boston University（波士顿大学） 纠错

专题命中 图文多模态 ：multimodal(abstract,comments)；image-text(abstract)；分类 cs.CV、cs.AI

AI总结 本文提出Mull-Tokens，一种无需模态切换的潜在令牌，用于空间、时间等多模态推理，通过预训练和微调在四个挑战性基准上实现3%-16%的提升。

Comments Project webpage: https://arijitray.com/multimodal_thinking/, Accepted to CVPR 2026 (Findings Track)

SIA: Enhancing Safety via Intent Awareness for Vision-Language Models

Youngjin Na, Sangheon Jeong, Youngwan Lee, Jian Lee, Dawoon Jeong, Youngman Kim

机构 * VLM Safety LAB, MODULABS（视觉语言模型安全实验室，MODULABS） ； ETRI（电子技术研究院） ； KAIST（韩国科学技术院） 纠错

专题命中 图文多模态 ：multimodal(abstract,comments)；image-text(abstract)；分类 cs.CV、cs.AI

Comments Accepted to Safe and Trustworthy Multimodal AI Systems(SafeMM-AI) Workshop at ICCV2025, Non-archival track

Underwater Camouflaged Object Tracking Meets Vision-Language SAM2

Chunhui Zhang, Li Liu, Guanjie Huang, Zhipeng Zhang, Hao Wen, Xi Zhou, Shiming Ge, Yanfeng Wang

机构 * Shanghai Jiao Tong University（上海交通大学） ； The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）） ； CloudWalk Technology Co., Ltd（云walk科技有限公司） ； Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所） ； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） 纠错

专题命中 图文多模态 ：multimodal(abstract,comments)；multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted to CVPR 2025 Workshop on CV4Animals. https://github.com/983632847/Awesome-Multimodal-Object-Tracking

COST: Contrastive One-Stage Transformer for Vision-Language Small Object Tracking

Chunhui Zhang, Li Liu, Jialin Gao, Xin Sun, Hao Wen, Xi Zhou, Shiming Ge, Yanfeng Wang

专题命中 图文多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI；multimodal(comments)

Comments Preprint submitted to Elsevier. https://github.com/983632847/Awesome-Multimodal-Object-Tracking

NYCU-TWO at Memotion 3: Good Foundation, Good Teacher, then you have Good Meme Analysis

Yu-Chien Tang, Kuang-Da Wang, Ting-Yun Ou, Wen-Chih Peng

专题命中 图文多模态 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL；multimodal(comments)

Comments De-Factify 2: Second Workshop on Multimodal Fact Checking and Hate Speech Detection, co-located with AAAI 2023

Shift-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment

用于微调CLIP的移位感知校准：利用图像-文本对齐

Song-Lin Lv, Yu-Yang Chen, Zhi Zhou, Lan-Zhe Guo

专题命中 图文多模态 ：image-text(title)；分类 cs.CV

AI总结 研究针对微调CLIP时预测置信度与准确性不一致问题，提出无需训练的移位感知校准（SAC）方法，利用原始与微调CLIP输出对数差异作校准信号，经实验验证该方法在多数据集和微调方法上提升了校准效果。

CAIRN: Cross-Room 3D Scene Understanding with Topology-Aware Large Multimodal Models

CAIRN：使用拓扑感知大型多模态模型进行跨房间3D场景理解

He Liang, Chenyang Ma, Yiming Zhang, Sangyun Shin, Andrew Markham, Niki Trigoni, Yuhang He

机构 * University of Oxford（牛津大学） ； Microsoft（微软公司） ； Simon Fraser University（西蒙弗雷泽大学） 纠错

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV

AI总结 研究针对现有3D-LLMs不能处理多房间场景问题，提出拓扑感知3D-LLM即CAIRN。其将Transformer注意力与场景层次对齐，通过多种方式增强模型能力。在新基准CAIRN-MR上实验，CAIRN在多房间任务中大幅超越前人，单房间任务也具竞争力。

Comments Project Page: https://oceansdepp.github.io/cairn_web/

BUS: Brain-Inspired Unsupervised Self-Reflection via Backward Prediction for Multimodal Reasoning

BUS：用于高级多模态推理的受脑启发的无监督自我反思

Jiacheng Yang, Tongying Xiao, Yunkai Dang, Cong Wang, Yuekun Yang, Qi Fan, Tianyu Ding, Wenbin Li, Feng Miao, Yang Gao

机构 * AAAI Press（美国人工智能协会出版社） 纠错

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV

AI总结 研究针对VLM处理复杂视觉任务的不足，受脑启发提出BUS无监督训练框架，通过反向预测提升其反思推理能力，无需标注数据，与多种微调方法兼容，经实验验证在多任务中有效提升了VLM推理性能。