多模态大模型

Multi-level Fusion of Wav2vec 2.0 and BERT for Multimodal Emotion Recognition

Zihan Zhao, Yanfeng Wang, Yu Wang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted to INTERSPEECH 2022

Brainish: Formalizing A Multimodal Language for Intelligence and Consciousness

Paul Pu Liang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Annual Meeting of the Association for the Scientific Study of Consciousness 2022

Multimodal Knowledge Alignment with Reinforcement Learning

Youngjae Yu, Jiwan Chung, Heeseung Yun, Jack Hessel, JaeSung Park, Ximing Lu, Prithviraj Ammanabrolu, Rowan Zellers, Ronan Le Bras, Gunhee Kim, Yejin Choi

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Multimodal Semi-Supervised Learning for Text Recognition

Aviad Aberdam, Roy Ganz, Shai Mazor, Ron Litman

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Code will be published upon publication

Multimodal Representation Learning With Text and Images

Aishwarya Jayagopal, Ankireddy Monica Aiswarya, Ankita Garg, Srinivasan Kolumam Nandakumar

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

A Comparative Analysis of Decision-Level Fusion for Multimodal Driver Behaviour Understanding

Alina Roitberg, Kunyu Peng, Zdravko Marinov, Constantin Seibold, David Schneider, Rainer Stiefelhagen

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted at Intelligent Vehicles Symposium 2022, IEEE

Training-Free Robust Multimodal Learning via Sample-Wise Jacobian Regularization

Zhengqi Gao, Sucheng Ren, Zihui Xue, Siting Li, Hang Zhao

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、eess.AS

M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining

Xiao Dong, Xunlin Zhan, Yangxin Wu, Yunchao Wei, Michael C. Kampffmeyer, Xiaoyong Wei, Minlong Lu, Yaowei Wang, Xiaodan Liang

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.MM

Comments CVPR2022

Sememe Prediction for BabelNet Synsets using Multilingual and Multimodal Information

Fanchao Qi, Chuancheng Lv, Zhiyuan Liu, Xiaojun Meng, Maosong Sun, Hai-Tao Zheng

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by Findings of ACL 2022 as a long paper. Camera-ready version

Multi-modal Graph Learning for Disease Prediction

Shuai Zheng, Zhenfeng Zhu, Zhizhe Liu, Zhenyu Guo, Yang Liu, Yuchen Yang, Yao Zhao

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments Published by IEEE TMI

A Multi-modal Fusion Framework Based on Multi-task Correlation Learning for Cancer Prognosis Prediction

Kaiwen Tan, Weixian Huang, Xiaofeng Liu, Jinlong Hu, Shoubin Dong

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Group Gated Fusion on Attention-based Bidirectional Alignment for Multimodal Emotion Recognition

Pengfei Liu, Kun Li, Helen Meng

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments Published in INTERSPEECH-2020

Journal ref INTERSPEECH 2020

MoCA: Incorporating Multi-stage Domain Pretraining and Cross-guided Multimodal Attention for Textbook Question Answering

Fangzhi Xu, Qika Lin, Jun Liu, Lingling Zhang, Tianzhe Zhao, Qi Chai, Yudai Pan

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.AI、cs.MM

Comments 9 pages, 6 figures

ScaleVLAD: Improving Multimodal Sentiment Analysis via Multi-Scale Fusion of Locally Descriptors

Huaishao Luo, Lei Ji, Yanyong Huang, Bin Wang, Shenggong Ji, Tianrui Li

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

StrucTexT: Structured Text Understanding with Multi-Modal Transformers

Yulin Li, Yuxi Qian, Yuchen Yu, Xiameng Qin, Chengquan Zhang, Yan Liu, Kun Yao, Junyu Han, Jingtuo Liu, Errui Ding

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Comments ACM Multimedia 2021. 9 pages

A Transformer-based Cross-modal Fusion Model with Adversarial Training for VQA Challenge 2021

Ke-Han Lu, Bo-Han Fang, Kuan-Yu Chen

专题命中 多模态训练与对齐 ：cross-modal(title,abstract)；分类 cs.CV、cs.CL

Comments CVPR 2021 Workshop: Visual Question Answering (VQA) Challenge

MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding

Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, Nicolas Carion

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records

Zhen Xu, David R. So, Andrew M. Dai

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted for publication at the Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21)

FiLMing Multimodal Sarcasm Detection with Attention

Sundesh Gupta, Aditya Shah, Miten Shah, Laribok Syiemlieh, Chandresh Maurya

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.MM

Comments 13 pages

Research on facial expression recognition based on Multimodal data fusion and neural network

Yi Han, Xubin Wang, Zhengyu Lu

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis

Wei Han, Hui Chen, Soujanya Poria

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted as a long paper at EMNLP 2021

A multimodal deep learning approach for named entity recognition from social media

Meysam Asgari-Chenaghlu, M. Reza Feizi-Derakhshi, Leili Farzinvash, M. A. Balafar, Cina Motamed

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.MM

Improving Multimodal fusion via Mutual Dependency Maximisation

Pierre Colombo, Emile Chapuis, Matthieu Labeau, Chloe Clavel

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Journal ref EMNLP 2021

UniT: Multimodal Multitask Learning with a Unified Transformer

Ronghang Hu, Amanpreet Singh

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments 16 pages

MMGCN: Multimodal Fusion via Deep Graph Convolution Network for Emotion Recognition in Conversation

Jingwen Hu, Yuchen Liu, Jinming Zhao, Qin Jin

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Data

Nathaniel Braman, Jacob W. H. Gordon, Emery T. Goossens, Caleb Willis, Martin C. Stumpe, Jagadish Venkataraman

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.MM

Comments Accepted for presentation at MICCAI 2021

Dizygotic Conditional Variational AutoEncoder for Multi-Modal and Partial Modality Absent Few-Shot Learning

Yi Zhang, Sheng Huang, Xi Peng, Dan Yang

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments 13 pages

A Review on Explainability in Multimodal Deep Neural Nets

Gargi Joshi, Rahee Walambe, Ketan Kotecha

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments 24 pages 6 figures

Journal ref in IEEE Access, vol. 9, pp. 59800-59821, 2021

Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

Aditya Prakash, Kashyap Chitta, Andreas Geiger

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments CVPR 2021

LAMPRET: Layout-Aware Multimodal PreTraining for Document Understanding

Te-Lin Wu, Cheng Li, Mingyang Zhang, Tao Chen, Spurthi Amba Hombaiah, Michael Bendersky

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL