多模态大模型

Noise-powered Multi-modal Knowledge Graph Representation Framework

Zhuo Chen, Yin Fang, Yichi Zhang, Lingbing Guo, Jiaoyan Chen, Jeff Z. Pan, Huajun Chen, Wen Zhang

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Comments COLING 2025 Accepted, Repo is available at https://github.com/zjukg/SNAG

A Heterogeneous Multimodal Graph Learning Framework for Recognizing User Emotions in Social Networks

Sree Bhattacharyya, Shuhua Yang, James Z. Wang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Imagine while Reasoning in Space: Multimodal Visualization-of-Thought

Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulić, Furu Wei

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments 11 pages, 6 figures, 4 tables (27 pages, 10 figures, 16 tables including references and appendices)

Dynamic Multimodal Fusion via Meta-Learning Towards Micro-Video Recommendation

Han Liu, Yinwei Wei, Fan Liu, Wenjie Wang, Liqiang Nie, Tat-Seng Chua

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.MM

Comments This paper has been accepted by ACM Transactions on Information Systems

Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models

Yifan Li, Hangyu Guo, Kun Zhou, Wayne Xin Zhao, Ji-Rong Wen

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments ECCV 2024 Oral

EAGLE: Enhanced Visual Grounding Minimizes Hallucinations in Instructional Multimodal Models

Andrés Villa, Juan León Alcázar, Motasem Alfarra, Vladimir Araujo, Alvaro Soto, Bernard Ghanem

专题命中 多模态训练与对齐 ：multimodal(title)；multi-modal(abstract)；分类 cs.CV、cs.AI

Comments 12 pages, 4 figures, 8 tables

Google is all you need: Semi-Supervised Transfer Learning Strategy For Light Multimodal Multi-Task Classification Model

Haixu Liu, Penghao Jiang, Zerui Tao

专题命中 多模态训练与对齐 ：multimodal(title)；multi-modal(abstract)；分类 cs.CV、cs.AI

Multimodal Fusion and Coherence Modeling for Video Topic Segmentation

Hai Yu, Chong Deng, Qinglin Zhang, Jiaqing Liu, Qian Chen, Wen Wang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models

Jieyu Zhang, Le Xue, Linxin Song, Jun Wang, Weikai Huang, Manli Shu, An Yan, Zixian Ma, Juan Carlos Niebles, Silvio Savarese, Caiming Xiong, Zeyuan Chen, Ranjay Krishna, Ran Xu

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments code: https://github.com/JieyuZ2/ProVision dataset: https://huggingface.co/datasets/Salesforce/ProVision-10M

COMO: Cross-Mamba Interaction and Offset-Guided Fusion for Multimodal Object Detection

Chang Liu, Xin Ma, Xiaochen Yang, Yuxiang Zhang, Yanni Dong

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation

Xiaofeng Zhang, Fanshuo Zeng, Yihao Quan, Zheng Hui, Jiawei Yao

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions

Mohammadmostafa Rostamkhani, Baktash Ansari, Hoorieh Sabzevari, Farzan Rahmani, Sauleh Eetemadi

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning

Neale Ratzlaff, Man Luo, Xin Su, Vasudev Lal, Phillip Howard

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

SurvMamba: State Space Model with Multi-grained Multi-modal Interaction for Survival Prediction

Ying Chen, Jiajing Xie, Yuxiang Lin, Yuhang Song, Wenxian Yang, Rongshan Yu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

What Differentiates Educational Literature? A Multimodal Fusion Approach of Transformers and Computational Linguistics

Jordan J. Bird

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

freePruner: A Training-free Approach for Large Multimodal Model Acceleration

Bingxin Xu, Yuzhang Shang, Yunhao Ge, Qian Lou, Yan Yan

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Unsupervised Homography Estimation on Multimodal Image Pair via Alternating Optimization

Sanghyeob Song, Jaihyun Lew, Hyemi Jang, Sungroh Yoon

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments This paper is accepted to the Thirty-Eighth Annual Conference on Neural Information Processing Systems (NeurIPS 2024)

Edge-Enhanced Dilated Residual Attention Network for Multimodal Medical Image Fusion

Meng Zhou, Yuxuan Zhang, Xiaolan Xu, Jiayi Wang, Farzad Khalvati

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments An extended version of the paper accepted at IEEE BIBM 2024

CMATH: Cross-Modality Augmented Transformer with Hierarchical Variational Distillation for Multimodal Emotion Recognition in Conversation

Xiaofei Zhu, Jiawei Cheng, Zhou Yang, Zhuo Chen, Qingyang Wang, Jianfeng Yao

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.MM

VMID: A Multimodal Fusion LLM Framework for Detecting and Identifying Misinformation of Short Videos

Weihao Zhong, Yinhao Xiao, Minghui Xu, Xiuzhen Cheng

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments arXiv admin note: text overlap with arXiv:2211.10973 by other authors

Toward Robust Incomplete Multimodal Sentiment Analysis via Hierarchical Representation Learning

Mingcheng Li, Dingkang Yang, Yang Liu, Shunli Wang, Jiawei Chen, Shuaibing Wang, Jinjie Wei, Yue Jiang, Qingyao Xu, Xiaolu Hou, Mingyang Sun, Ziyun Qian, Dongliang Kou, Lihua Zhang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted by NeurIPS 2024

EEG-based Multimodal Representation Learning for Emotion Recognition

Kang Yin, Hye-Bin Shin, Dan Li, Seong-Whan Lee

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning

Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, Alexander Hauptmann

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.AI、cs.MM

Comments Accepted at NeurIPS 2024. 49 pages, 13 figures, Project: https://github.com/ZebangCheng/Emotion-LLaMA, Demo: https://huggingface.co/spaces/ZebangCheng/Emotion-LLaMA

Parameter-Efficient Fine-Tuning Medical Multimodal Large Language Models for Medical Visual Grounding

Jinlong He, Pengfei Li, Gang Liu, Shenjun Zhong

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

M$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning

Taowen Wang, Yiyang Liu, James Chenhao Liang, junhan zhao, Yiming Cui, Yuning Mao, Shaoliang Nie, Jiahao Liu, Fuli Feng, Zenglin Xu, Cheng Han, Lifu Huang, Qifan Wang, Dongfang Liu

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2024

No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance

Vishaal Udandarao, Ameya Prabhu, Adhiraj Ghosh, Yash Sharma, Philip H. S. Torr, Adel Bibi, Samuel Albanie, Matthias Bethge

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Short version accepted at DPFM, ICLR'24; Full paper at NeurIPS'24

AMPLE: Emotion-Aware Multimodal Fusion Prompt Learning for Fake News Detection

Xiaoman Xu, Xiangrun Li, Taihang Wang, Ye Jiang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Rate

Qidong Huang, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Jiaqi Wang, Dahua Lin, Weiming Zhang, Nenghai Yu

专题命中 多模态训练与对齐 ：cross-modal(title)；multi-modal(abstract)；分类 cs.CV、cs.CL

Comments Project page: https://github.com/shikiw/Modality-Integration-Rate

ATLAS: Adapter-Based Multi-Modal Continual Learning with a Two-Stage Learning Strategy

Hong Li, Zhiquan Tan, Xingyu Li, Weiran Huang

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Learning Content-Aware Multi-Modal Joint Input Pruning via Bird's-Eye-View Representation

Yuxin Li, Yiheng Li, Xulei Yang, Mengying Yu, Zihang Huang, Xiaojun Wu, Chai Kiat Yeo

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI