多模态大模型

E5-V: Universal Embeddings with Multimodal Large Language Models

Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Code and models are available at https://github.com/kongds/E5-V

Application of Multimodal Fusion Deep Learning Model in Disease Recognition

Xiaoyi Liu, Hongjie Qiu, Muqing Li, Zhou Yu, Yutian Yang, Yafeng Yan

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Cognitive Visual-Language Mapper: Advancing Multimodal Comprehension with Enhanced Visual Knowledge Alignment

Yunxin Li, Xinyu Chen, Baotian Hu, Haoyuan Shi, Min Zhang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments 12 pages,4 figures; Accepted by ACL 2024 Main Conference

Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in Autonomous Driving

Yang Lou, Qun Song, Qian Xu, Rui Tan, Jianping Wang

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments In proceedings of the 26th European Conference on Artificial Intelligence ECAI 2023. 8 pages + 2 appendix pages

Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning

Chong Ma, Hanqi Jiang, Wenting Chen, Yiwei Li, Zihao Wu, Xiaowei Yu, Zhengliang Liu, Lei Guo, Dajiang Zhu, Tuo Zhang, Dinggang Shen, Tianming Liu, Xiang Li

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments 12 pages, 6 figures

CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models

Junho Kim, Hyunjun Kim, Yeonju Kim, Yong Man Ro

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Project page: https://ivy-lvlm.github.io/CODE/

On the Limits of Multi-modal Meta-Learning with Auxiliary Task Modulation Using Conditional Batch Normalization

Jordi Armengol-Estapé, Vincent Michalski, Ramnath Kumar, Pierre-Luc St-Charles, Doina Precup, Samira Ebrahimi Kahou

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion

Zehan Wang, Ziang Zhang, Xize Cheng, Rongjie Huang, Luping Liu, Zhenhui Ye, Haifeng Huang, Yang Zhao, Tao Jin, Peng Gao, Zhou Zhao

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ICML 2024. The code and checkpoints will be released at https://github.com/zehanwang01/FreeBind

EventLens: Leveraging Event-Aware Pretraining and Cross-modal Linking Enhances Visual Commonsense Reasoning

Mingjie Ma, Zhihuan Yu, Yichao Ma, Guohui Li

专题命中 多模态训练与对齐 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL

Cooperative Sentiment Agents for Multimodal Sentiment Analysis

Shanmin Wang, Hui Shuai, Qingshan Liu, Fei Wang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Data-Efficient Multimodal Fusion on a Single GPU

Noël Vouitsis, Zhaoyan Liu, Satya Krishna Gorti, Valentin Villecroze, Jesse C. Cresswell, Guangwei Yu, Gabriel Loaiza-Ganem, Maksims Volkovs

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments CVPR 2024 (Highlight)

Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts

Jialin Wu, Xia Hu, Yaqing Wang, Bo Pang, Radu Soricut

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

TelME: Teacher-leading Multimodal Fusion Network for Emotion Recognition in Conversation

Taeyang Yun, Hyunkuk Lim, Jeonghwan Lee, Min Song

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Comments NAACL 2024 main conference

FSMR: A Feature Swapping Multi-modal Reasoning Approach with Joint Textual and Visual Clues

Shuang Li, Jiahua Wang, Lijie Wen

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

A Survey of IMU Based Cross-Modal Transfer Learning in Human Activity Recognition

Abhi Kamboj, Minh Do

专题命中 多模态训练与对齐 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Enhancing Multimodal Entity and Relation Extraction with Variational Information Bottleneck

Shiyao Cui, Jiangxia Cao, Xin Cong, Jiawei Sheng, Quangang Li, Tingwen Liu, Jinqiao Shi

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CL、cs.MM

Journal ref IEEE/ACM Transactions on Audio, Speech and Language Processing, 2024

SyCoCa: Symmetrizing Contrastive Captioners with Attentive Masking for Multimodal Alignment

Ziping Ma, Furong Xu, Jian Liu, Ming Yang, Qingpei Guo

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Hierarchical Aligned Multimodal Learning for NER on Tweet Posts

Peipei Liu, Hong Li, Yimo Ren, Jie Liu, Shuaizong Si, Hongsong Zhu, Limin Sun

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL、cs.MM

Frequency Spectrum is More Effective for Multimodal Representation and Fusion: A Multimodal Spectrum Rumor Detector

An Lao, Qi Zhang, Chongyang Shi, Longbing Cao, Kun Yi, Liang Hu, Duoqian Miao

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI、cs.MM

Comments 12 pages, AAAI-2024

Investigating the Catastrophic Forgetting in Multimodal Large Language Models

Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, Yi Ma

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL、cs.AI

Geodesic Multi-Modal Mixup for Robust Fine-Tuning

Changdae Oh, Junhyuk So, Hoyoon Byun, YongTaek Lim, Minchul Shin, Jong-June Jeon, Kyungwoo Song

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL

Comments To appear at NeurIPS 2023

An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models

Yadong Lu, Chunyuan Li, Haotian Liu, Jianwei Yang, Jianfeng Gao, Yelong Shen

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

Comments Released at LLaVA Model Zoo: https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md

4DRVO-Net: Deep 4D Radar-Visual Odometry Using Multi-Modal and Multi-Scale Adaptive Fusion

Guirong Zhuo, Shouyi Lu, Huanyu Zhou, Lianqing Zheng, Lu Xiong

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments 14 pages,12 figures

Multi-Grained Multimodal Interaction Network for Entity Linking

Pengfei Luo, Tong Xu, Shiwei Wu, Chen Zhu, Linli Xu, Enhong Chen

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by KDD 2023

Modality Influence in Multimodal Machine Learning

Abdelhamid Haouhat, Slimane Bellaouar, Attia Nehar, Hadda Cherroun

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CL、cs.AI

Comments 10 pages

Fusion for Visual-Infrared Person ReID in Real-World Surveillance Using Corrupted Multimodal Data

Arthur Josi, Mahdi Alehdaghi, Rafael M. O. Cruz, Eric Granger

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments 31 pages, 11 figures, First version submitted to IJCV journal

Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining

Bingqian Lin, Zicong Chen, Mingjie Li, Haokun Lin, Hang Xu, Yi Zhu, Jianzhuang Liu, Wenjia Cai, Lei Yang, Shen Zhao, Chenfei Wu, Ling Chen, Xiaojun Chang, Yi Yang, Lei Xing, Xiaodan Liang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Project page: https://github.com/chenzcv7/MOTOR

Learning Transferable Pedestrian Representation from Multimodal Information Supervision

Liping Bao, Longhui Wei, Xiaoyu Qiu, Wengang Zhou, Houqiang Li, Qi Tian

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Explaining Multimodal Data Fusion: Occlusion Analysis for Wilderness Mapping

Burak Ekim, Michael Schmitt

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Comments 5 pages, 2 figures

Masked Vision and Language Modeling for Multi-modal Representation Learning

Gukyeong Kwon, Zhaowei Cai, Avinash Ravichandran, Erhan Bas, Rahul Bhotika, Stefano Soatto

专题命中 多模态训练与对齐 ：multi-modal(title)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments International Conference on Learning Representations (ICLR) 2023