多模态大模型

BEACON: Balancing Convenience and Nutrition in Meals With Long-Term Group Recommendations and Reasoning on Multimodal Recipes

Vansh Nagpal, Siva Likitha Valluru, Kausik Lakkaraju, Biplav Srivastava

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 6 pages (including references), 1 figure, 2 tables

Xmodel-VLM: A Simple Baseline for Multimodal Vision Language Model

Wanting Xu, Yang Liu, Langping He, Xucheng Huang, Ling Jiang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detection

Shruti Palaskar, Oggi Rudovic, Sameer Dharur, Florian Pesce, Gautam Krishna, Aswin Sivaraman, Jack Berkowitz, Ahmed Hussen Abdelaziz, Saurabh Adya, Ahmed Tewfik

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted at Interspeech 2024

Towards Robust Instruction Tuning on Multimodal Large Language Models

Wei Han, Hui Chen, Soujanya Poria

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 24 pages, 7 figures

Multimodal Belief Prediction

John Murzaku, Adil Soubki, Owen Rambow

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments John Murzaku and Adil Soubki contributed equally to this work

Journal ref Interspeech 2024

Mixture of Rationale: Multi-Modal Reasoning Mixture for Visual Question Answering

Tao Li, Linjun Shou, Xuejun Liu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

II-MMR: Identifying and Improving Multi-modal Multi-hop Reasoning in Visual Question Answering

Jihyung Kil, Farideh Tavazoee, Dongyeop Kang, Joo-Kyung Kim

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted to ACL 2024 Findings

Efficient Ensemble for Multimodal Punctuation Restoration using Time-Delay Neural Network

Xing Yi Liu, Homayoon Beigi

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments 6 pages, 1 figure, 5 tables, paper at IMCOM 2024, technical report at Recognition Technologies, Inc

Journal ref 2024 18th International Conference on Ubiquitous Information Management and Communication (IMCOM), 2024, pp. 1-6

Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models

Yue Zhang, Hehe Fan, Yi Yang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Distilling Privileged Multimodal Information for Expression Recognition using Optimal Transport

Muhammad Haseeb Aslam, Muhammad Osama Zeeshan, Soufiane Belharbi, Marco Pedersoli, Alessandro Koerich, Simon Bacon, Eric Granger

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

A review of deep learning-based information fusion techniques for multimodal medical image classification

Yihao Li, Mostafa El Habib Daho, Pierre-Henri Conze, Rachid Zeghlache, Hugo Le Boité, Ramin Tadayoni, Béatrice Cochener, Mathieu Lamard, Gwenolé Quellec

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Quantifying and Enhancing Multi-modal Robustness with Modality Preference

Zequn Yang, Yake Wei, Ce Liang, Di Hu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.MM

Comments Accepted to ICLR 2024

Exploring the Transferability of Visual Prompting for Multimodal Large Language Models

Yichi Zhang, Yinpeng Dong, Siyuan Zhang, Tianzan Min, Hang Su, Jun Zhu

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted in CVPR 2024 as Poster (Highlight)

Soft-Prompting with Graph-of-Thought for Multi-modal Representation Learning

Juncheng Yang, Zuchao Li, Shuai Xie, Wei Yu, Shijun Li, Bo Du

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Comments This paper is accepted to LREC-COLING 2024

DELAN: Dual-Level Alignment for Vision-and-Language Navigation by Cross-Modal Contrastive Learning

Mengfei Du, Binhao Wu, Jiwen Zhang, Zhihao Fan, Zejun Li, Ruipu Luo, Xuanjing Huang, Zhongyu Wei

专题命中 多模态训练与对齐 ：cross-modal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted by LREC-COLING 2024

An Evolutionary Network Architecture Search Framework with Adaptive Multimodal Fusion for Hand Gesture Recognition

Yizhang Xia, Shihao Song, Zhanglu Hou, Junwen Xu, Juan Zou, Yuan Liu, Shengxiang Yang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Models

Minjie Zhu, Yichen Zhu, Xin Liu, Ning Liu, Zhiyuan Xu, Chaomin Shen, Yaxin Peng, Zhicai Ou, Feifei Feng, Jian Tang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Multimodal Fusion with Pre-Trained Model Features in Affective Behaviour Analysis In-the-wild

Zhuofan Wen, Fengyu Zhang, Siyuan Zhang, Haiyang Sun, Mingyu Xu, Licai Sun, Zheng Lian, Bin Liu, Jianhua Tao

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Multimodal Fusion Method with Spatiotemporal Sequences and Relationship Learning for Valence-Arousal Estimation

Jun Yu, Gongpeng Zhao, Yongqi Wang, Zhihong Wei, Yang Zheng, Zerui Zhang, Zhongpeng Cai, Guochen Xie, Jichao Zhu, Wangyuan Zhu

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、eess.AS

Comments 8 pages,3 figures

M2DA: Multi-Modal Fusion Transformer Incorporating Driver Attention for Autonomous Driving

Dongyang Xu, Haokun Li, Qingfan Wang, Ziying Song, Lei Chen, Hanming Deng

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Shared and Private Information Learning in Multimodal Sentiment Analysis with Deep Modal Alignment and Self-supervised Multi-Task Learning

Songning Lai, Jiakang Li, Guinan Guo, Xifeng Hu, Yulong Li, Yuan Tan, Zichen Song, Yutong Liu, Zhaoxia Ren, Chun Wan, Danmin Miao, Zhi Liu

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Journal ref International Joint Conference on Neural Networks (IJCNN) 2024

Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification

Pingping Zhang, Yuhao Wang, Yang Liu, Zhengzheng Tu, Huchuan Lu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.MM

Comments This work is accepted by CVPR2024. More modifications may be performed

Multimodal Fusion of EHR in Structures and Semantics: Integrating Clinical Records and Notes with Hypergraph and LLM

Hejie Cui, Xinyu Fang, Ran Xu, Xuan Kan, Joyce C. Ho, Carl Yang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

A Multimodal Intermediate Fusion Network with Manifold Learning for Stress Detection

Morteza Bodaghi, Majid Hosseini, Raju Gottumukkala

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments This work was accepted to The 3rd International Conference on Computing and Machine Intelligence (ICMI 2024)

MOAB: Multi-Modal Outer Arithmetic Block For Fusion Of Histopathological Images And Genetic Data For Brain Tumor Grading

Omnia Alwazzan, Abbas Khan, Ioannis Patras, Gregory Slabaugh

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Journal ref pages={1--5},year={2023},organization={IEEE}

Multimodal Fusion of EMG and Vision for Human Grasp Intent Inference in Prosthetic Hand Control

Mehrshad Zandigohar, Mo Han, Mohammadreza Sharif, Sezen Yagmur Gunay, Mariusz P. Furmanek, Mathew Yarossi, Paolo Bonato, Cagdas Onal, Taskin Padir, Deniz Erdogmus, Gunar Schirner

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Journal ref Front. Robot. AI 11 (2024) Sec. Biomedical Robotics

GeoDecoder: Empowering Multimodal Map Understanding

Feng Qi, Mian Dai, Zixian Zheng, Chao Wang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Review of multimodal machine learning approaches in healthcare

Felix Krones, Umar Marikkar, Guy Parsons, Adam Szmul, Adam Mahdi

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments 5 figures, 5 tables

OCT-SelfNet: A Self-Supervised Framework with Multi-Modal Datasets for Generalized and Robust Retinal Disease Detection

Fatema-E Jannat, Sina Gholami, Minhaj Nur Alam, Hamed Tabkhi

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments 12 pages, 7 figures, 6 tables

A Generalized Multi-Modal Fusion Detection Framework

Leichao Cui, Xiuxian Li, Min Meng, Xiaoyu Mo

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI