多模态大模型 - arXivDaily 专题

2307.03591 2023-07-10 cs.AI cs.IR 79%

Structure Guided Multi-modal Pre-trained Transformer for Knowledge Graph Reasoning

Ke Liang, Sihang Zhou, Yue Liu, Lingyuan Meng, Meng Liu, Xinwang Liu

专题命中多模态训练与对齐：multi-modal(title)；multimodal(abstract)；分类 cs.AI

Comments This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessed

2307.03425 2023-07-10 cs.CV 79%

Registration-Free Hybrid Learning Empowers Simple Multimodal Imaging System for High-quality Fusion Detection

Yinghan Guan, Haoran Dai, Zekuan Yu, Shouyu Wang, Yuanjie Gu

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV

2303.00865 2023-07-06 cs.CV 79%

AMIGO: Sparse Multi-Modal Graph Transformer with Shared-Context Processing for Representation Learning of Giga-pixel Images

Ramin Nakhli, Puria Azadi Moghadam, Haoyang Mi, Hossein Farahani, Alexander Baras, Blake Gilks, Ali Bashashati

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV

Comments Accepted at CVPR 2023

2307.01486 2023-07-06 eess.IV cs.CV 79%

H-DenseFormer: An Efficient Hybrid Densely Connected Transformer for Multimodal Tumor Segmentation

Jun Shi, Hongyu Kan, Shulan Ruan, Ziqi Zhu, Minfan Zhao, Liang Qiao, Zhaohui Wang, Hong An, Xudong Xue

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV

Comments 11 pages, 2 figures. This paper has been accepted by Medical Image Computing and Computer-Assisted Intervention(MICCAI) 2023

2306.16991 2023-06-30 cs.CV 79%

Integrating Large Pre-trained Models into Multimodal Named Entity Recognition with Evidential Fusion

Weide Liu, Xiaoyang Zhong, Jingwen Hou, Shaohua Li, Haozhe Huang, Yuming Fang

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV

2306.15765 2023-06-29 cs.CV 79%

A Novel Two Stream Decision Level Fusion of Vision and Inertial Sensors Data for Automatic Multimodal Human Activity Recognition System

Santosh Kumar Yadav, Muhtashim Rafiqi, Egna Praneeth Gummana, Kamlesh Tiwari, Hari Mohan Pandey, Shaik Ali Akbara

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV

2306.11746 2023-06-22 cs.SI cs.MM 79%

Focusing on Relevant Responses for Multi-modal Rumor Detection

Jun Li, Yi Bin, Liang Peng, Yang Yang, Yangyang Li, Hao Jin, Zi Huang

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.MM

Comments Submitted to TKDE

2203.16952 2023-06-21 cs.CV cs.LG eess.IV 79%

Multimodal Fusion Transformer for Remote Sensing Image Classification

Swalpa Kumar Roy, Ankur Deria, Danfeng Hong, Behnood Rasti, Antonio Plaza, Jocelyn Chanussot

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV

Comments Published in IEEE Transactions on Geoscience and Remote Sensing

2306.11020 2023-06-21 cs.CL 79%

Dual-Gated Fusion with Prefix-Tuning for Multi-Modal Relation Extraction

Qian Li, Shu Guo, Cheng Ji, Xutan Peng, Shiyao Cui, Jianxin Li

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CL

2306.10146 2023-06-21 cs.CV eess.IV 79%

Multi-task 3D building understanding with multi-modal pretraining

Shicheng Xu

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV

Comments 8 pages, 9 figures, 9 tables

2212.05265 2023-06-21 cs.CV 79%

Multi-Sem Fusion: Multimodal Semantic Fusion for 3D Object Detection

Shaoqing Xu, Fang Li, Ziying Song, Jin Fang, Sifen Wang, Zhi-Xin Yang

专题命中多模态训练与对齐：multimodal(title)；multi-modal(abstract)；分类 cs.CV

2207.03128 2023-06-16 cs.CV 79%

PointMCD: Boosting Deep Point Cloud Encoders via Multi-view Cross-modal Distillation for 3D Shape Recognition

Qijian Zhang, Junhui Hou, Yue Qian

专题命中多模态训练与对齐：cross-modal(title,abstract)；分类 cs.CV

Comments Accepted to TMM

2204.12423 2023-06-16 cs.LG cs.CV 79%

RadioPathomics: Multimodal Learning in Non-Small Cell Lung Cancer for Adaptive Radiotherapy

Matteo Tortora, Ermanno Cordelli, Rosa Sicilia, Lorenzo Nibid, Edy Ippolito, Giuseppe Perrone, Sara Ramella, Paolo Soda

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV

2212.10773 2023-06-13 cs.CL 79%

MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning

Zhiyang Xu, Ying Shen, Lifu Huang

专题命中多模态训练与对齐：multi-modal(title)；multimodal(abstract)；分类 cs.CL

Comments ACL 2023, dataset url: https://github.com/VT-NLP/MultiInstruct

2305.18721 2023-06-12 cs.CV 79%

LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding

Yi Tu, Ya Guo, Huan Chen, Jinyang Tang

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV

Comments Accepted by ACL 2023 main conference

2306.02673 2023-06-06 eess.IV cs.CV cs.LG 79%

Cross-Modal Vertical Federated Learning for MRI Reconstruction

Yunlu Yan, Hong Wang, Yawen Huang, Nanjun He, Lei Zhu, Yuexiang Li, Yong Xu, Yefeng Zheng

专题命中多模态训练与对齐：cross-modal(title,abstract)；分类 cs.CV

Comments 12 pages, 7 figures

2306.01523 2023-06-05 cs.CV cs.LG 79%

Transformer-based Multi-Modal Learning for Multi Label Remote Sensing Image Classification

David Hoffmann, Kai Norman Clasen, Begüm Demir

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV

Comments Accepted at IEEE International Geoscience and Remote Sensing Symposium 2023

2306.00792 2023-06-02 cs.CV 79%

Learning Across Decentralized Multi-Modal Remote Sensing Archives with Federated Learning

Barış Büyüktaş, Gencer Sumbul, Begüm Demir

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV

Comments Accepted at IEEE International Geoscience and Remote Sensing Symposium (IGARSS) 2023. Our code is available at https://git.tu-berlin.de/rsim/MM-FL

2304.03916 2023-06-01 cs.LG cs.AI 79%

Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning

Yu Yang, Besmira Nushi, Hamid Palangi, Baharan Mirzasoleiman

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.AI

2305.11439 2023-05-22 cs.CV 79%

Few-Shot Learning with Visual Distribution Calibration and Cross-Modal Distribution Alignment

Runqi Wang, Hao Zheng, Xiaoyue Duan, Jianzhuang Liu, Yuning Lu, Tian Wang, Songcen Xu, Baochang Zhang

专题命中多模态训练与对齐：cross-modal(title,abstract)；分类 cs.CV

2210.14616 2023-05-16 cs.CR cs.AI 79%

A Late Multi-Modal Fusion Model for Detecting Hybrid Spam E-mail

Zhibo Zhang, Ernesto Damiani, Hussam Al Hamadi, Chan Yeob Yeun, Fatma Taher

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.AI

Comments The content of this paper needs to be updated

Journal ref Index in journal International Journal of Computer Theory and Engineering (IJCTE), 2023

2212.01923 2023-05-11 cs.DB cs.AI cs.LG 79%

Query-Driven Knowledge Base Completion using Multimodal Path Fusion over Multimodal Knowledge Graph

Yang Peng, Daisy Zhe Wang

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.AI

2206.06488 2023-05-11 cs.CV cs.LG 79%

Multimodal Learning with Transformers: A Survey

Peng Xu, Xiatian Zhu, David A. Clifton

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV

Comments This paper is accepted by IEEE TPAMI

2208.10240 2023-05-10 cs.CL 79%

A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction

Weimin Lyu, Xinyu Dong, Rachel Wong, Songzhu Zheng, Kayley Abell-Hart, Fusheng Wang, Chao Chen

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CL

Comments AMIA Annual Symposium Proceedings 2022

Journal ref AMIA Annu Symp Proc 2022

2305.03810 2023-05-09 cs.HC cs.CV 79%

Distilled Mid-Fusion Transformer Networks for Multi-Modal Human Activity Recognition

Jingcheng Li, Lina Yao, Binghao Li, Claude Sammut

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV

Comments 13 pages, 6 figures

2211.07098 2023-05-09 cs.AI cs.DB cs.LG 79%

Knowledge Base Completion using Web-Based Question Answering and Multimodal Fusion

Yang Peng, Daisy Zhe Wang

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.AI

2209.06954 2023-05-09 cs.CV 79%

Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answering

Jingjing Jiang, Ziyi Liu, Nanning Zheng

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV

Comments 20 pages, 6 figures, 13 tables

2305.00201 2023-05-02 cs.CV 79%

Instruction-ViT: Multi-Modal Prompts for Instruction Learning in ViT

Zhenxiang Xiao, Yuzhong Chen, Lu Zhang, Junjie Yao, Zihao Wu, Xiaowei Yu, Yi Pan, Lin Zhao, Chong Ma, Xinyu Liu, Wei Liu, Xiang Li, Yixuan Yuan, Dinggang Shen, Dajiang Zhu, Tianming Liu, Xi Jiang

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV

2304.05351 2023-05-01 cs.CL cs.LG q-fin.ST 79%

The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges

Qianqian Xie, Weiguang Han, Yanzhao Lai, Min Peng, Jimin Huang

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CL

Comments 13 pages

2304.11979 2023-04-25 cs.IR cs.MM 79%

Attention-guided Multi-step Fusion: A Hierarchical Fusion Network for Multimodal Recommendation

Yan Zhou, Jie Guo, Hao Sun, Bin Song, Fei Richard Yu

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.MM

1. 多模态训练与对齐 6918 篇

Structure Guided Multi-modal Pre-trained Transformer for Knowledge Graph Reasoning

Registration-Free Hybrid Learning Empowers Simple Multimodal Imaging System for High-quality Fusion Detection

AMIGO: Sparse Multi-Modal Graph Transformer with Shared-Context Processing for Representation Learning of Giga-pixel Images

H-DenseFormer: An Efficient Hybrid Densely Connected Transformer for Multimodal Tumor Segmentation

Integrating Large Pre-trained Models into Multimodal Named Entity Recognition with Evidential Fusion

A Novel Two Stream Decision Level Fusion of Vision and Inertial Sensors Data for Automatic Multimodal Human Activity Recognition System

Focusing on Relevant Responses for Multi-modal Rumor Detection

Multimodal Fusion Transformer for Remote Sensing Image Classification

Dual-Gated Fusion with Prefix-Tuning for Multi-Modal Relation Extraction

Multi-task 3D building understanding with multi-modal pretraining

Multi-Sem Fusion: Multimodal Semantic Fusion for 3D Object Detection

PointMCD: Boosting Deep Point Cloud Encoders via Multi-view Cross-modal Distillation for 3D Shape Recognition

RadioPathomics: Multimodal Learning in Non-Small Cell Lung Cancer for Adaptive Radiotherapy

MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning

LayoutMask: Enhance Text-Layout Interaction in Multi-modal Pre-training for Document Understanding

Cross-Modal Vertical Federated Learning for MRI Reconstruction

Transformer-based Multi-Modal Learning for Multi Label Remote Sensing Image Classification

Learning Across Decentralized Multi-Modal Remote Sensing Archives with Federated Learning

Mitigating Spurious Correlations in Multi-modal Models during Fine-tuning

Few-Shot Learning with Visual Distribution Calibration and Cross-Modal Distribution Alignment

A Late Multi-Modal Fusion Model for Detecting Hybrid Spam E-mail

Query-Driven Knowledge Base Completion using Multimodal Path Fusion over Multimodal Knowledge Graph

Multimodal Learning with Transformers: A Survey

A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction

Distilled Mid-Fusion Transformer Networks for Multi-Modal Human Activity Recognition

Knowledge Base Completion using Web-Based Question Answering and Multimodal Fusion

Correlation Information Bottleneck: Towards Adapting Pretrained Multimodal Models for Robust Visual Question Answering

Instruction-ViT: Multi-Modal Prompts for Instruction Learning in ViT

The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges

Attention-guided Multi-step Fusion: A Hierarchical Fusion Network for Multimodal Recommendation