多模态大模型 - arXivDaily 专题

2506.18042 2025-06-24 cs.CV 83%

CmFNet: Cross-modal Fusion Network for Weakly-supervised Segmentation of Medical Images

Dongdong Meng, Sheng Li, Hao Wu, Suqing Tian, Wenjun Ma, Guoping Wang, Xueqing Yan

机构 * School of Physics, Peking University（北京大学物理学院）； School of Computer Science, Peking University（北京大学计算机学院）； Department of Radiotherapy, Peking University Cancer Hospital（北京大学肿瘤医院放疗科）； Department of Radiation Oncology, Peking University Third Hospital（北京大学第三医院放疗科）

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments 10 pages, 6 figures

2506.11976 2025-06-24 cs.CV cs.LG 83%

How Visual Representations Map to Language Feature Space in Multimodal LLMs

Constantin Venhoff, Ashkan Khakzar, Sonia Joseph, Philip Torr, Neel Nanda

机构 * University of Oxford（牛津大学）； McGill University（麦吉尔大学）； Meta

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2405.08454 2025-06-24 cs.CL 83%

Alignment Helps Make the Most of Multimodal Data

Christian Arnold, Andreas Küpfer

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL

Comments Working Paper

2506.16737 2025-06-23 cs.CV 83%

Cross-modal Offset-guided Dynamic Alignment and Fusion for Weakly Aligned UAV Object Detection

Liu Zongzhen, Luo Hui, Wang Zhixing, Wei Yuxing, Zuo Haorui, Zhang Jianlin

机构 * State Key Laboratory of Optical Field Manipulation Science and Technology, Chinese Academy of Sciences（光学场操控科学与技术国家重点实验室，中国科学院）； Institute of Optics and Electronics, Chinese Academy of Sciences（中国科学院光电研究所）

专题命中多模态训练与对齐：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

2505.07895 2025-06-23 cs.LG cs.AI 83%

Representation Learning with Mutual Influence of Modalities for Node Classification in Multi-Modal Heterogeneous Networks

Jiafan Li, Jiaqi Zhu, Liang Chang, Yilin Li, Miaomiao Li, Yang Wang, Hongan Wang

机构 * Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）； University of Chinese Academy of Sciences（中国科学院大学）； School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）； Binzhou Institute of Technology（滨州职业技术学院）

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.AI

2506.14532 2025-06-18 cs.CL 83%

M2BeamLLM: Multimodal Sensing-empowered mmWave Beam Prediction with Large Language Models

Can Zheng, Jiguang He, Chung G. Kang, Guofa Cai, Zitong Yu, Merouane Debbah

机构 * Department of Electrical and Computer Engineering, Korea University（韩国大学电子与计算机工程系）； School of Computing and Information Technology, Great Bay University（大湾大学计算机与信息科技学院）； Dongguan Key Laboratory for Intelligence and Information Technology（东莞智能与信息科技重点实验室）； Great Bay Institute for Advanced Study (GBIAS)（大湾先进研究学院）； School of Information Engineering, Guangdong University of Technology（广东工业大学信息工程学院）； Center for 6G Technology, Khalifa University of Science and Technology（哈里发科技大学6G技术中心）

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CL

Comments 13 pages, 20 figures

2503.07950 2025-06-17 cs.CV 83%

Decoupled Cross-Modal Alignment Network for Text-RGBT Person Retrieval and A High-Quality Benchmark

Yifei Deng, Chenglong Li, Zhenyu Chen, Zihen Xu, Jin Tang

机构 * School of Computer Science and Technology, Anhui University（安徽大学计算机科学与技术学院）； School of Artificial Intelligence, Anhui University（安徽大学人工智能学院）； National Key Laboratory of Opto-Electronic Information Acquisition and Protection Technology（光电信息采集与防护技术国家重点实验室）； Anhui Provincial Key Laboratory of Multimodal Cognitive Computation（安徽省多模态认知计算重点实验室）

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

2503.00513 2025-06-17 cs.CV 83%

Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning

Hanxun Yu, Wentong Li, Song Wang, Junbo Chen, Jianke Zhu

机构 * Zhejiang University（浙江大学）； Nanjing University of Aeronautics and Astronautics（南京航空航天大学）

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments CVPR2025, Code Link: https://github.com/hanxunyu/Inst3D-LMM

2411.18688 2025-06-17 cs.CR cs.AI cs.LG 83%

Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment

Soumya Suvra Ghosal, Souradip Chakraborty, Vaibhav Singh, Tianrui Guan, Mengdi Wang, Alvaro Velasquez, Ahmad Beirami, Furong Huang, Dinesh Manocha, Amrit Singh Bedi

机构 * University of Maryland（马里兰大学）； Indian Institute of Technology Bombay（印度班加罗尔理工学院）； Princeton University（普林斯顿大学）； University of Colorado Boulder（科罗拉多大学博尔德分校）； Capital One（Capital One公司）； University of Central Florida（佛罗里达中央大学）

专题命中多模态训练与对齐：multi-modal(title)；multimodal(abstract)；MLLM(abstract)；分类 cs.AI

Comments Accepted to CVPR 2025

2308.10019 2025-06-17 cs.CV 83%

Dissecting RGB-D Learning for Improved Multi-modal Fusion

Hao Chen, Haoran Zhou, Yunshu Zhang, Zheng Lin, Yongjian Deng

机构 * School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）； Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及跨学科应用国家重点实验室（东南大学））； Tsinghua University（清华大学）； College of Computer Science, Beijing University of Technology（北京理工大学计算机学院）

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2506.11691 2025-06-16 cs.CV 83%

DMAF-Net: An Effective Modality Rebalancing Framework for Incomplete Multi-Modal Medical Image Segmentation

Libin Lan, Hongxing Li, Zunhui Xia, Yudong Zhang

机构 * College of Computer Science and Engineering, Chongqing University of Technology（重庆理工大学计算机科学与工程学院）； School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments 12 pages, 4 figures, 3 tables

2506.11672 2025-06-16 cs.CV 83%

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning

Chendi Ge, Xin Wang, Zeyang Zhang, Hong Chen, Jiapei Fan, Longtao Huang, Hui Xue, Wenwu Zhu

机构 * Department of Computer Science and Technology, BNRist, Tsinghua University, Beijing, China（计算机科学与技术系，BNRist，清华大学，北京，中国）； Alibaba Group, Hangzhou, China（阿里巴巴集团，杭州，中国）

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments Accepted by ICML 2025

2506.07138 2025-06-10 cs.CV cs.AI cs.CL cs.MM 83%

Learning Compact Vision Tokens for Efficient Large Multimodal Models

Hao Tang, Chengchao Shen

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments The source code and trained weights are available at https://github.com/visresearch/LLaVA-STF

2506.02308 2025-06-10 cs.LG cs.AI 83%

MINT: Multimodal Instruction Tuning with Multimodal Interaction Grouping

Xiaojun Shan, Qi Cao, Xing Han, Haofei Yu, Paul Pu Liang

机构 * University of California, San Diego（加州大学圣地亚哥分校）； Johns Hopkins University（约翰霍普金斯大学）； University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）； Massachusetts Institute of Technology（麻省理工学院）

专题命中多模态训练与对齐：multimodal(title,abstract)；multimodal foundation model(abstract)；分类 cs.AI

2403.04588 2025-06-05 cs.AI 83%

Zero-shot cross-modal transfer of Reinforcement Learning policies through a Global Workspace

Léopold Maytié, Benjamin Devillers, Alexandre Arnold, Rufin VanRullen

机构 * Airbus AI Research（空客人工智能研究）

专题命中多模态训练与对齐：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.AI

Journal ref Reinforcement Learning Journal, Vol. 3, 2024, pp. 1410-1426

2505.24176 2025-06-02 cs.MM 83%

ISMAF: Intrinsic-Social Modality Alignment and Fusion for Multimodal Rumor Detection

Zihao Yu, Xiang Li, Jing Zhang

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.MM

2505.23365 2025-05-30 cs.CV 83%

MCFNet: A Multimodal Collaborative Fusion Network for Fine-Grained Semantic Classification

Yang Qiao, Xiaoyu Zhong, Xiaofeng Gu, Zhiguo Yu

机构 * School of Integrated Circuits, Jiangnan University, Wuxi 214122, China（集成电路学院，江南大学，无锡214122，中国）

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2503.12941 2025-05-30 cs.CL cs.LG 83%

HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model

Haiyang Guo, Fanhu Zeng, Ziwei Xiang, Fei Zhu, Da-Han Wang, Xu-Yao Zhang, Cheng-Lin Liu

机构 * School of Advanced Interdisciplinary Sciences, UCAS（UCAS先进交叉科学学院）； MAIS, CASIA（CASIA人工智能研究所）； School of Artificial Intelligence, UCAS（UCAS人工智能学院）； Centre for Artificial Intelligence and Robotics, HKISI-CAS（HKISI-CAS人工智能与机器人中心）； FKLPRIU, School of Computer and Information Engineering, Xiamen University of Technology（厦门理工学院计算机与信息工程学院）

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL

Comments ACL 2025 (Main)

2505.21079 2025-05-28 cs.CV 83%

Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts

Yue Zhang, Yingzhao Jian, Hehe Fan, Yi Yang, Roger Zimmermann

机构 * Zhejiang University（浙江大学）； National University of Singapore（新加坡国立大学）

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

2412.15576 2025-05-28 cs.RO cs.CV 83%

QUART-Online: Latency-Free Large Multimodal Language Model for Quadruped Robot Learning

Xinyang Tong, Pengxiang Ding, Yiguo Fan, Donglin Wang, Wenjie Zhang, Can Cui, Mingyang Sun, Han Zhao, Hongyin Zhang, Yonghao Dang, Siteng Huang, Shangke Lyu

机构 * MiLAB, Westlake University, Hangzhou, 310030, China（西交利物浦大学微实验室，杭州，310030，中国）； Zhejiang University, Hangzhou, 310027, China（浙江大学，杭州，310027，中国）； Beijing University of Posts and Telecommunications, Beijing, 100876, China（北京邮电大学，北京，100876，中国）

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments Accepted to ICRA 2025; Github page: https://quart-online.github.io

2505.16916 2025-05-23 cs.CR cs.CV 83%

Backdoor Cleaning without External Guidance in MLLM Fine-tuning

Xuankun Rong, Wenke Huang, Jian Liang, Jinhe Bi, Xun Xiao, Yiming Li, Bo Du, Mang Ye

专题命中多模态训练与对齐：MLLM(title)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

2505.13828 2025-05-21 cs.AI 83%

Multimodal RAG-driven Anomaly Detection and Classification in Laser Powder Bed Fusion using Large Language Models

Kiarash Naghavi Khanghah, Zhiling Chen, Lela Romeo, Qian Yang, Rajiv Malhotra, Farhad Imani, Hongyi Xu

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.AI

Comments ASME 2025 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference IDETC/CIE2025, August 17-20, 2025, Anaheim, CA (IDETC2025-168615)

2505.12693 2025-05-20 cs.CV 83%

TACOcc:Target-Adaptive Cross-Modal Fusion with Volume Rendering for 3D Semantic Occupancy

Luyao Lei, Shuo Xu, Yifan Bai, Xing Wei

机构 * School of Software Engineering（软件工程学院）； Xi’an Jiaotong University（西安交通大学）

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

2505.12251 2025-05-20 cs.CV 83%

SMFusion: Semantic-Preserving Fusion of Multimodal Medical Images for Enhanced Clinical Diagnosis

Haozhe Xiang, Han Zhang, Yu Cheng, Xiongwen Quan, Wanwan Huang

机构 * College of Information and Intelligence, Hunan Agricultural University（湖南农业大学信息与智能学院）； College of Artificial Intelligence, Nankai University（南开大学人工智能学院）

专题命中多模态训练与对齐：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

2411.17141 2025-05-19 cs.CV 83%

Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation

Xu Zheng, Haiwei Xue, Jialei Chen, Yibo Yan, Lutao Jiang, Yuanhuiyi Lyu, Kailun Yang, Linfeng Zhang, Xuming Hu

机构 * HKUST(GZ)（香港科技大学（广州））； Tsinghua University（清华大学）； Nagoya University（名古屋大学）； Hunan University（湖南大学）； Shanghai Jiao Tong University（上海交通大学）

专题命中多模态训练与对齐：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments Preprint

2505.10105 2025-05-16 cs.RO cs.AI 83%

EmbodiedMAE: A Unified 3D Multi-Modal Representation for Robot Manipulation

Zibin Dong, Fei Ni, Yifu Yuan, Yinchuan Li, Jianye Hao

机构 * Tianjin University（天津大学）； Huawei Noah’s Ark Lab（华为诺亚实验室）

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.AI

2410.03577 2025-05-09 cs.CV 83%

Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models

Xin Zou, Yizhou Wang, Yibo Yan, Yuanhuiyi Lyu, Kening Zheng, Sirui Huang, Junkai Chen, Peijie Jiang, Jia Liu, Chang Tang, Xuming Hu

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments Accepted by ICML 2025

2504.19002 2025-04-29 cs.LG cs.CV cs.RO 83%

Deep Learning-Based Multi-Modal Fusion for Robust Robot Perception and Navigation

Delun Lai, Yeyubei Zhang, Yunchong Liu, Chaojie Li, Huadong Mo

专题命中多模态训练与对齐：multi-modal(title)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments 6 pages, 4 figures

2504.15624 2025-04-29 cs.CV 83%

FaceInsight: A Multimodal Large Language Model for Face Perception

Jingzhi Li, Changjiang Luo, Ruoyu Chen, Hua Zhang, Wenqi Ren, Jianhou Gan, Xiaochun Cao

机构 * Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）； School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）； School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区计算机科学与技术学院）； Key Laboratory of Education Informatization for Nationalities (Yunnan Normal University), Ministry of Education（民族教育信息化重点实验室（云南师范大学））

专题命中多模态训练与对齐：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

2504.17261 2025-04-25 cs.LG cs.AI 83%

Symbolic Representation for Any-to-Any Generative Tasks

Jiaqi Chen, Xiaoye Zhu, Yue Wang, Tianyang Liu, Xinhui Chen, Ying Chen, Chak Tou Leong, Yifei Ke, Joseph Liu, Yiwen Yuan, Julian McAuley, Li-jia Li

机构 * Stanford University（斯坦福大学）； Fellou AI（Fellou人工智能）； Fudan University（复旦大学）； South China University of Technology（华南理工大学）； Cornell University（康奈尔大学）； University of California San Diego（加州大学圣地亚哥分校）； Wuhan University（武汉大学）； University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）； Hong Kong Polytechnic University（香港理工大学）； University of Southern California（南加州大学）； Carnegie Mellon University（卡内基梅隆大学）； LiveX AI（LiveX人工智能）

专题命中多模态训练与对齐：any-to-any(title)；multimodal(abstract)；cross-modal(abstract)；分类 cs.AI

1. 多模态训练与对齐 6903 篇

CmFNet: Cross-modal Fusion Network for Weakly-supervised Segmentation of Medical Images

How Visual Representations Map to Language Feature Space in Multimodal LLMs

Alignment Helps Make the Most of Multimodal Data

Cross-modal Offset-guided Dynamic Alignment and Fusion for Weakly Aligned UAV Object Detection

Representation Learning with Mutual Influence of Modalities for Node Classification in Multi-Modal Heterogeneous Networks

M2BeamLLM: Multimodal Sensing-empowered mmWave Beam Prediction with Large Language Models

Decoupled Cross-Modal Alignment Network for Text-RGBT Person Retrieval and A High-Quality Benchmark

Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning

Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment

Dissecting RGB-D Learning for Improved Multi-modal Fusion

DMAF-Net: An Effective Modality Rebalancing Framework for Incomplete Multi-Modal Medical Image Segmentation

Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning

Learning Compact Vision Tokens for Efficient Large Multimodal Models

MINT: Multimodal Instruction Tuning with Multimodal Interaction Grouping

Zero-shot cross-modal transfer of Reinforcement Learning policies through a Global Workspace

ISMAF: Intrinsic-Social Modality Alignment and Fusion for Multimodal Rumor Detection

MCFNet: A Multimodal Collaborative Fusion Network for Fine-Grained Semantic Classification

HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model

Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts

QUART-Online: Latency-Free Large Multimodal Language Model for Quadruped Robot Learning

Backdoor Cleaning without External Guidance in MLLM Fine-tuning

Multimodal RAG-driven Anomaly Detection and Classification in Laser Powder Bed Fusion using Large Language Models

TACOcc:Target-Adaptive Cross-Modal Fusion with Volume Rendering for 3D Semantic Occupancy

SMFusion: Semantic-Preserving Fusion of Multimodal Medical Images for Enhanced Clinical Diagnosis

Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation

EmbodiedMAE: A Unified 3D Multi-Modal Representation for Robot Manipulation

Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models

Deep Learning-Based Multi-Modal Fusion for Robust Robot Perception and Navigation

FaceInsight: A Multimodal Large Language Model for Face Perception

Symbolic Representation for Any-to-Any Generative Tasks