多模态大模型 - arXivDaily 专题

2503.13709 2025-03-19 cs.LG 82%

Multi-modal Time Series Analysis: A Tutorial and Survey

Yushan Jiang, Kanghui Ning, Zijie Pan, Xuyang Shen, Jingchao Ni, Wenchao Yu, Anderson Schneider, Haifeng Chen, Yuriy Nevmyvaka, Dongjin Song

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)

2503.13383 2025-03-18 cs.CV cs.AI cs.CL cs.LG 82%

Cream of the Crop: Harvesting Rich, Scalable and Transferable Multi-Modal Data for Instruction Fine-Tuning

Mengyao Lyu, Yan Li, Huasong Zhong, Wenhao Yang, Hui Chen, Jungong Han, Guiguang Ding, Zhenheng Yang

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments update comparison with sota and analysis

2503.10726 2025-03-17 cs.LG eess.IV 82%

Prototype-Guided Cross-Modal Knowledge Enhancement for Adaptive Survival Prediction

Fengchun Liu, Linghan Cai, Zhikang Wang, Zhiyuan Fan, Jin-gang Yu, Hao Chen, Yongbing Zhang

专题命中多模态训练与对齐：cross-modal(title,abstract)；multimodal(abstract)

2503.09652 2025-03-14 eess.IV 82%

4D-ACFNet: A 4D Attention Mechanism-Based Prognostic Framework for Colorectal Cancer Liver Metastasis Integrating Multimodal Spatiotemporal Features

Zesheng Li, Wei Yang, Yan Su, Yiran Zhu, Yuhan Tang, Haoran Chen, Chengchang Pan, Honggang Qi

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)

Comments 8 pages,6 figures,2 tables,submitted to the 33rd ACM International Conference on Multimedia(ACM MM 2025)

2503.06313 2025-03-11 cs.CV cs.AI cs.CL cs.LG cs.RO 82%

Advancing Autonomous Vehicle Intelligence: Deep Learning and Multimodal LLM for Traffic Sign Recognition and Robust Lane Detection

Chandan Kumar Sah, Ankit Kumar Shaw, Xiaoli Lian, Arsalan Shahid Baig, Tuopu Wen, Kun Jiang, Mengmeng Yang, Diange Yang

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 11 pages, 9 figures

2503.00037 2025-03-04 cs.CL cs.AI cs.CV cs.LG 82%

Zero-Shot Defense Against Toxic Images via Inherent Multimodal Alignment in LVLMs

Wei Zhao, Zhe Li, Yige Li, Jun Sun

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2501.03895 2025-03-04 cs.CV cs.AI cs.CL 82%

LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token

Shaolei Zhang, Qingkai Fang, Zhe Yang, Yang Feng

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to ICLR 2025. Code: https://github.com/ictnlp/LLaVA-Mini Model: https://huggingface.co/ICTNLP/llava-mini-llama-3.1-8b

2407.11712 2025-02-04 cs.IR 82%

Fine-tuning Multimodal Large Language Models for Product Bundling

Xiaohao Liu, Jie Wu, Zhulin Tao, Yunshan Ma, Yinwei Wei, Tat-seng Chua

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)

Comments Accepted by KDD 2025 (CR)

2501.16400 2025-01-29 eess.IV physics.med-ph 82%

CSF-Net: Cross-Modal Spatiotemporal Fusion Network for Pulmonary Nodule Malignancy Predicting

Yin Shen, Zhaojie Fang, Ke Zhuang, Guanyu Zhou, Xiao Yu, Yucheng Zhao, Yuan Tian, Ruiquan Ge, Changmiao Wang, Xiaopeng Fan, Ahmed Elazab

专题命中多模态训练与对齐：cross-modal(title,abstract)；multimodal(abstract)

Comments This paper has been accepted by the 2025 IEEE International Symposium on Biomedical Imaging

2407.21033 2025-01-28 cs.IR cs.AI cs.CL cs.CV 82%

Multi-Grained Query-Guided Set Prediction Network for Grounded Multimodal Named Entity Recognition

Jielong Tang, Zhenxing Wang, Ziyang Gong, Jianxing Yu, Xiangwei Zhu, Jian Yin

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 13 pages, 7 figures

2501.11538 2025-01-22 cs.LG 82%

DenoMAE: A Multimodal Autoencoder for Denoising Modulation Signals

Atik Faysal, Taha Boushine, Mohammad Rostami, Reihaneh Gh. Roshan, Huaxia Wang, Nikhil Muralidhar, Avimanyu Sahoo, Yu-Dong Yao

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)

2412.16243 2024-12-24 cs.LG 82%

Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Data

Zhiqiang Tang, Zihan Zhong, Tong He, Gerald Friedland

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)

2406.15334 2024-12-23 cs.CV cs.AI cs.CL cs.LG 82%

Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning

Brandon Huang, Chancharik Mitra, Assaf Arbelle, Leonid Karlinsky, Trevor Darrell, Roei Herzig

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Published in NeurIPS 2024

2412.14536 2024-12-20 q-bio.NC 82%

Multi-Modal Latent Variables for Cross-Individual Primary Visual Cortex Modeling and Analysis

Yu Zhu, Bo Lei, Chunfeng Song, Wanli Ouyang, Shan Yu, Tiejun Huang

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)

Comments AAAI 2025

2412.04903 2024-12-17 cs.CV cs.AI cs.CL cs.LG 82%

EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation

Yongxin Wang, Meng Cao, Haokun Lin, Mingfei Han, Liang Ma, Jin Jiang, Yuhao Cheng, Xiaodan Liang

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 19 pages

2410.16081 2024-11-19 astro-ph.SR astro-ph.GA astro-ph.IM physics.comp-ph physics.data-an 82%

Deep Multimodal Representation Learning for Stellar Spectra

Tobias Buck, Christian Schwarz

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)

Comments accepted to the Machine Learning and the Physical Sciences Workshop, NeurIPS 2024

2411.00029 2024-11-04 cs.CL cs.AI cs.CV cs.LG 82%

Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models

Donghoon Kim, Gusang Lee, Kyuhong Shim, Byonghyo Shim

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Findings of EMNLP 2024

2404.01595 2024-10-30 cs.LG stat.ME stat.ML 82%

Propensity Score Alignment of Unpaired Multimodal Data

Johnny Xi, Jana Osea, Zuheng Xu, Jason Hartford

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)

Comments NeurIPS 2024

2103.01296 2024-10-28 cs.RO 82%

Learning Multimodal Contact-Rich Skills from Demonstrations Without Reward Engineering

Mythra V. Balakuntala, Upinder Kaur, Xin Ma, Juan Wachs, Richard M. Voyles

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)

Comments Accepted at IEEE ICRA 2021. This work has been submitted to the IEEE for possible publication

2407.02623 2024-10-15 cs.CY cs.AI cs.CL cs.CV 82%

Uplifting Lower-Income Data: Strategies for Socioeconomic Perspective Shifts in Large Multi-modal Models

Joan Nwatu, Oana Ignat, Rada Mihalcea

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2410.04491 2024-10-08 cs.CL cs.AI cs.MM 82%

Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis

Xinyu Feng, Yuming Lin, Lihua He, You Li, Liang Chang, Ya Zhou

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Accepted to EMNLP Findings 2024

2406.11839 2024-10-08 cs.CV cs.AI cs.CL cs.LG 82%

mDPO: Conditional Preference Optimization for Multimodal Large Language Models

Fei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu, Sheng Zhang, Hoifung Poon, Muhao Chen

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to EMNLP 2024 Main Conference. Project website: https://feiwang96.github.io/mDPO

2008.06682 2024-10-08 eess.AS cs.AI cs.CL cs.SD 82%

Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition

Shamane Siriwardhana, Andrew Reis, Rivindu Weerasekera, Suranga Nanayakkara

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted to INTERSPEECH 2020

2407.10385 2024-10-01 cs.CL cs.AI cs.CV cs.LG 82%

By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting

Hyungjun Yoon, Biniyam Aschalew Tolera, Taesik Gong, Kimin Lee, Sung-Ju Lee

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024) Main

2406.05766 2024-09-24 cs.LG cs.AI cs.CL cs.CV 82%

Set-CLIP: Exploring Aligned Semantic From Low-Alignment Multimodal Data Through A Distribution View

Zijia Song, Zelin Zang, Yelin Wang, Guozheng Yang, Kaicheng yu, Wanyu Chen, Miaoyu Wang, Stan Z. Li

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2408.12321 2024-08-27 cs.CL cs.CV cs.MM 82%

MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model

Chaoya Jiang, Jia Hongrui, Haiyang Xu, Wei Ye, Mengfan Dong, Ming Yan, Ji Zhang, Fei Huang, Shikun Zhang

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

2408.10500 2024-08-23 cs.MM cs.CV cs.SD eess.AS 82%

SZTU-CMU at MER2024: Improving Emotion-LLaMA with Conv-Attention for Multimodal Emotion Recognition

Zebang Cheng, Shuyuan Tu, Dawei Huang, Minghan Li, Xiaojiang Peng, Zhi-Qi Cheng, Alexander G. Hauptmann

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Ranked 1st in MER24@IJCAI and MRAC24@ACM MM (MER-NOISE & MER-OV (self-evaluated))

2408.09798 2024-08-20 cs.LG 82%

Enhance Modality Robustness in Text-Centric Multimodal Alignment with Adversarial Prompting

Yun-Da Tsai, Ting-Yu Yen, Keng-Te Liao, Shou-De Lin

专题命中多模态训练与对齐：multimodal(title,abstract)；multimodal foundation model(abstract)

Comments arXiv admin note: text overlap with arXiv:2407.05036

2407.21391 2024-08-01 cs.SD cs.CV cs.MM eess.AS 82%

Design and Development of Laughter Recognition System Based on Multimodal Fusion and Deep Learning

Fuzheng Zhao, Yu Bai

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 7 pages,2 figures

2405.17430 2024-07-30 cs.CV cs.AI cs.CL cs.LG 82%

Matryoshka Multimodal Models

Mu Cai, Jianwei Yang, Jianfeng Gao, Yong Jae Lee

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project Page: https://matryoshka-mm.github.io/

1. 多模态训练与对齐 6903 篇

Multi-modal Time Series Analysis: A Tutorial and Survey

Cream of the Crop: Harvesting Rich, Scalable and Transferable Multi-Modal Data for Instruction Fine-Tuning

Prototype-Guided Cross-Modal Knowledge Enhancement for Adaptive Survival Prediction

4D-ACFNet: A 4D Attention Mechanism-Based Prognostic Framework for Colorectal Cancer Liver Metastasis Integrating Multimodal Spatiotemporal Features

Advancing Autonomous Vehicle Intelligence: Deep Learning and Multimodal LLM for Traffic Sign Recognition and Robust Lane Detection

Zero-Shot Defense Against Toxic Images via Inherent Multimodal Alignment in LVLMs

LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token

Fine-tuning Multimodal Large Language Models for Product Bundling

CSF-Net: Cross-Modal Spatiotemporal Fusion Network for Pulmonary Nodule Malignancy Predicting

Multi-Grained Query-Guided Set Prediction Network for Grounded Multimodal Named Entity Recognition

DenoMAE: A Multimodal Autoencoder for Denoising Modulation Signals

Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Data

Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning

Multi-Modal Latent Variables for Cross-Individual Primary Visual Cortex Modeling and Analysis

EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation

Deep Multimodal Representation Learning for Stellar Spectra

Preserving Pre-trained Representation Space: On Effectiveness of Prefix-tuning for Large Multi-modal Models

Propensity Score Alignment of Unpaired Multimodal Data

Learning Multimodal Contact-Rich Skills from Demonstrations Without Reward Engineering

Uplifting Lower-Income Data: Strategies for Socioeconomic Perspective Shifts in Large Multi-modal Models

Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis

mDPO: Conditional Preference Optimization for Multimodal Large Language Models

Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognition

By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting

Set-CLIP: Exploring Aligned Semantic From Low-Alignment Multimodal Data Through A Distribution View

MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model

SZTU-CMU at MER2024: Improving Emotion-LLaMA with Conv-Attention for Multimodal Emotion Recognition

Enhance Modality Robustness in Text-Centric Multimodal Alignment with Adversarial Prompting

Design and Development of Laughter Recognition System Based on Multimodal Fusion and Deep Learning

Matryoshka Multimodal Models