多模态大模型 - arXivDaily 专题

2407.12438 2024-07-18 cs.LG 82%

Semantic-Aware Representation of Multi-Modal Data for Data Ingress: A Literature Review

Pierre Lamart, Yinan Yu, Christian Berger

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)

Comments Accepted at the 50th Euromicro Conference on Software Engineering and Advanced Applications (SEAA) 2024 as a short paper

2304.08658 2024-06-25 physics.app-ph cs.LG 82%

In-situ surface porosity prediction in DED (directed energy deposition) printed SS316L parts using multimodal sensor fusion

Adithyaa Karthikeyan, Himanshu Balhara, Abhishek Hanchate, Andreas K Lianos, Satish TS Bukkapatnam

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)

2406.06048 2024-06-17 cs.CV cs.AI cs.MM 82%

Robust Latent Representation Tuning for Image-text Classification

Hao Sun, Yu Song

专题命中多模态训练与对齐：image-text(title)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

2406.00017 2024-06-14 cs.CL cs.AI cs.MM 82%

PTA: Enhancing Multimodal Sentiment Analysis through Pipelined Prediction and Translation-based Alignment

Shezheng Song, Shasha Li, Shan Zhao, Chengyu Wang, Xiaopeng Li, Jie Yu, Qian Wan, Jun Ma, Tianwei Yan, Wentao Ma, Xiaoguang Mao

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Code will be released upon publication

2304.05600 2024-06-11 cs.SD cs.CV cs.LG cs.MM eess.AS 82%

Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning

Nikhil Singh, Chih-Wei Wu, Iroro Orife, Mahdi Kalayeh

专题命中多模态训练与对齐：cross-modal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to CVPR 2024

2404.04818 2024-04-11 cs.AI cs.CL cs.CV 82%

DWE+: Dual-Way Matching Enhanced Framework for Multimodal Entity Linking

Shezheng Song, Shasha Li, Shan Zhao, Xiaopeng Li, Chengyu Wang, Jie Yu, Jun Ma, Tianwei Yan, Bin Ji, Xiaoguang Mao

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments under review on TOIS. arXiv admin note: substantial text overlap with arXiv:2312.11816

2404.03171 2024-04-05 cs.SE cs.LG cs.PL 82%

Multi-modal Learning for WebAssembly Reverse Engineering

Hanxian Huang, Jishen Zhao

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)

Comments Accepted by ISSTA '24

2311.17076 2024-04-02 cs.CV cs.AI cs.CL cs.LG 82%

Compositional Chain-of-Thought Prompting for Large Multimodal Models

Chancharik Mitra, Brandon Huang, Trevor Darrell, Roei Herzig

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2403.13537 2024-03-21 cs.CL cs.AI cs.CV cs.LG 82%

What explains the success of cross-modal fine-tuning with ORCA?

Paloma García-de-Herreros, Vagrant Gautam, Philipp Slusallek, Dietrich Klakow, Marius Mosbach

专题命中多模态训练与对齐：cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2403.10279 2024-03-18 cs.CY 82%

Emotion-Aware Multimodal Fusion for Meme Emotion Detection

Shivam Sharma, Ramaneswaran S, Md. Shad Akhtar, Tanmoy Chakraborty

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)

Comments Accepted to IEEE Transactions on Affective Computing

2402.19407 2024-03-01 cs.IR 82%

MENTOR: Multi-level Self-supervised Learning for Multimodal Recommendation

Jinfeng Xu, Zheyu Chen, Shuo Yang, Jinze Li, Hewei Wang, Edith C. -H. Ngai

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)

2402.15444 2024-02-26 cs.AI cs.CL cs.LG cs.MM 82%

Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion

Yichi Zhang, Zhuo Chen, Lei Liang, Huajun Chen, Wen Zhang

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Accepted by LREC-COLING 2024

2402.05294 2024-02-09 cs.LG cs.AI cs.CL cs.CV 82%

Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detection

Pramit Saha, Divyanshu Mishra, Felix Wagner, Konstantinos Kamnitsas, J. Alison Noble

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 42 pages

2401.06827 2024-01-24 cs.CV cs.AI cs.CL 82%

APLe: Token-Wise Adaptive for Multi-Modal Prompt Learning

Guiming Cao, Kaize Shi, Hong Fu, Huaiwen Zhang, Guandong Xu

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 7 pages,3 figures

2312.10386 2023-12-19 cs.LG 82%

RedCore: Relative Advantage Aware Cross-modal Representation Learning for Missing Modalities with Imbalanced Missing Rates

Jun Sun, Xinxin Zhang, Shoukang Han, Yu-ping Ruan, Taihao Li

专题命中多模态训练与对齐：cross-modal(title,abstract)；multimodal(abstract)

2312.03231 2023-12-07 cs.LG cs.AI cs.CV cs.HC eess.AS 82%

Deep Multimodal Fusion for Surgical Feedback Classification

Rafal Kocielnik, Elyssa Y. Wong, Timothy N. Chu, Lydia Lin, De-An Huang, Jiayun Wang, Anima Anandkumar, Andrew J. Hung

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.AI、eess.AS

Journal ref Published in Proceedings of Machine Learning for Health 2024

2305.13583 2023-11-14 cs.CL cs.MM eess.AS eess.IV 82%

Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition

Yaoting Wang, Yuanchao Li, Paul Pu Liang, Louis-Philippe Morency, Peter Bell, Catherine Lai

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments *First two authors contributed equally

2310.16822 2023-10-26 cs.CL cs.AI cs.MM 82%

Prompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extraction

Xuming Hu, Junzhe Chen, Aiwei Liu, Shiao Meng, Lijie Wen, Philip S. Yu

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Accepted to ACM Multimedia 2023

2309.14398 2023-09-28 cs.LG cs.AI cs.CL eess.AS 82%

Seeing and hearing what has not been said; A multimodal client behavior classifier in Motivational Interviewing with interpretable fusion

Lucie Galland, Catherine Pelachaud, Florian Pecune

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 9 pages, 7 figures

2205.11501 2023-09-18 cs.CV cs.AI cs.CL 82%

VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering

Yanan Wang, Michihiro Yasunaga, Hongyu Ren, Shinya Wada, Jure Leskovec

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at ICCV 2023

2309.07120 2023-09-14 cs.CL cs.AI cs.CV cs.CY cs.LG 82%

Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics

Haoqin Tu, Bingchen Zhao, Chen Wei, Cihang Xie

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2309.02106 2023-09-06 cs.CL cs.AI cs.LG eess.AS 82%

Leveraging Label Information for Multimodal Emotion Recognition

Peiying Wang, Sunlu Zeng, Junqing Chen, Lu Fan, Meng Chen, Youzheng Wu, Xiaodong He

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted by Interspeech 2023

2307.16210 2023-08-02 cs.AI cs.CV cs.LG cs.MM 82%

Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment

Zhuo Chen, Lingbing Guo, Yin Fang, Yichi Zhang, Jiaoyan Chen, Jeff Z. Pan, Yangning Li, Huajun Chen, Wen Zhang

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments International Semantic Web Conference '23 (ISWC 2023), https://github.com/zjukg/UMAEA

2307.16745 2023-08-01 cs.CV cs.AI cs.CY cs.MM 82%

Advancing Smart Malnutrition Monitoring: A Multi-Modal Learning Approach for Vital Health Parameter Estimation

Ashish Marisetty, Prathistith Raj M, Praneeth Nemani, Venkanna Udutalapally, Debanjan Das

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

2307.05390 2023-07-12 cond-mat.mtrl-sci cs.LG 82%

CrysMMNet: Multimodal Representation for Crystal Property Prediction

Kishalay Das, Pawan Goyal, Seung-Cheol Lee, Satadeep Bhattacharjee, Niloy Ganguly

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)

Comments 14 pages, 4 fifures

2307.02761 2023-07-07 cs.IR 82%

Cross-Modal Content Inference and Feature Enrichment for Cold-Start Recommendation

Haokai Ma, Zhuang Qi, Xinxin Dong, Xiangxian Li, Yuze Zheng, Xiangxu Mengand Lei Meng

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)

Comments Accepted by International Joint Conference on Neural Networks 2023 (IJCNN)

2306.02137 2023-06-21 cs.LG 82%

Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detection

Mengzhu Sun, Xi Zhang, Jianqiang Ma, Sihong Xie, Yazheng Liu, Philip S. Yu

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)

Journal ref IEEE Transactions on Knowledge and Data Engineering, 2023

2209.05722 2023-05-18 cs.RO 82%

GrASPE: Graph based Multimodal Fusion for Robot Navigation in Unstructured Outdoor Environments

Kasun Weerakoon, Adarsh Jagan Sathyamoorthy, Jing Liang, Tianrui Guan, Utsav Patel, Dinesh Manocha

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)

2304.08028 2023-04-18 cs.CV cs.AI cs.MM 82%

MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learning

Shicai Wei, Yang Luo, Chunbo Luo

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 10 pages, 3 figures, CVPR2023

2209.15182 2023-04-12 cs.HC cs.RO 82%

Husformer: A Multi-Modal Transformer for Multi-Modal Human State Recognition

Ruiqi Wang, Wonse Jo, Dezhong Zhao, Weizheng Wang, Baijian Yang, Guohua Chen, Byung-Cheol Min

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)

1. 多模态训练与对齐 6903 篇

Semantic-Aware Representation of Multi-Modal Data for Data Ingress: A Literature Review

In-situ surface porosity prediction in DED (directed energy deposition) printed SS316L parts using multimodal sensor fusion

Robust Latent Representation Tuning for Image-text Classification

PTA: Enhancing Multimodal Sentiment Analysis through Pipelined Prediction and Translation-based Alignment

Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learning

DWE+: Dual-Way Matching Enhanced Framework for Multimodal Entity Linking

Multi-modal Learning for WebAssembly Reverse Engineering

Compositional Chain-of-Thought Prompting for Large Multimodal Models

What explains the success of cross-modal fine-tuning with ORCA?

Emotion-Aware Multimodal Fusion for Meme Emotion Detection

MENTOR: Multi-level Self-supervised Learning for Multimodal Recommendation

Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion

Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detection

APLe: Token-Wise Adaptive for Multi-Modal Prompt Learning

RedCore: Relative Advantage Aware Cross-modal Representation Learning for Missing Modalities with Imbalanced Missing Rates

Deep Multimodal Fusion for Surgical Feedback Classification

Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition

Prompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extraction

Seeing and hearing what has not been said; A multimodal client behavior classifier in Motivational Interviewing with interpretable fusion

VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering

Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics

Leveraging Label Information for Multimodal Emotion Recognition

Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment

Advancing Smart Malnutrition Monitoring: A Multi-Modal Learning Approach for Vital Health Parameter Estimation

CrysMMNet: Multimodal Representation for Crystal Property Prediction

Cross-Modal Content Inference and Feature Enrichment for Cold-Start Recommendation

Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detection

GrASPE: Graph based Multimodal Fusion for Robot Navigation in Unstructured Outdoor Environments

MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learning

Husformer: A Multi-Modal Transformer for Multi-Modal Human State Recognition