多模态大模型

MusicAIR: A Multimodal AI Music Generation Framework Powered by an Algorithm-Driven Core

MusicAIR: 一种由算法驱动核心的多模态AI音乐生成框架

Callie C. Liao, Duoduo Liao, Ellie L. Zhang

机构 * Stanford University（斯坦福大学） ； George Mason University（乔治·马歇尔大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

AI总结 MusicAIR通过算法驱动的核心生成音乐，能从歌词、文本和图像生成符合音乐理论的旋律谱，提升音乐创作效率并降低入门门槛。

Comments Accepted by IEEE Big Data 2025

MMVA: Multimodal Matching Based on Valence and Arousal across Images, Music, and Musical Captions

MMVA: 基于估值和唤醒度的多模态匹配

Suhwan Choi, Kyu Won Kim, Myungjoo Kang

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI、cs.MM、eess.AS

AI总结 MMVA通过多模态匹配基于估值和唤醒度，实现跨图像、音乐和音乐字幕的情感内容捕捉，并在估值-唤醒度预测任务中取得最佳性能。

Comments Paper accepted in Artificial Intelligence for Music workshop at AAAI 2025

ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism

Zedong Liu, Shenggan Cheng, Guangming Tan, Yang You, Dingwen Tao

机构 * Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所） ； University of Electronic Science and Technology of China（电子科技大学） ； National University of Singapore（新加坡国立大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；MLLM(abstract)

Comments Accepted at NeurIPS 2025 Oral (Thirty-Ninth Conference on Neural Information Processing Systems)

Audio-Visual Traffic Light State Detection for Urban Robots

Sagar Gupta, Akansel Cosgun

专题命中 音频语音多模态 ：audio-visual(title)；multimodal(abstract)；multi-modal(abstract)

Comments Submitted to IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2024

Journal ref 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

HCFSLN: Adaptive Hyperbolic Few-Shot Learning for Multimodal Anxiety Detection

Aditya Sneh, Nilesh Kumar Sahu, Anushka Sanjay Shelke, Arya Adyasha, Haroon R. Lone

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)

ELEGANCE: Efficient LLM Guidance for Audio-Visual Target Speech Extraction

Wenxuan Wu, Shuai Wang, Xixin Wu, Helen Meng, Haizhou Li

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CL、cs.MM、eess.AS

UTI-LLM: A Personalized Articulatory-Speech Therapy Assistance System Based on Multimodal Large Language Model

Yudong Yang, Xiaokang Liu, Shaofeng zhao, Rongfeng Su, Nan Yan, Lan Wang

机构 * Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China（深圳先进技术研究院，中国科学院，中国） ； Key Laboratory of Biomedical Imaging Science and System, Chinese Academy of Sciences, China（生物医学成像科学与系统重点实验室，中国科学院，中国） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；MLLM(abstract)

MEXA: Towards General Multimodal Reasoning with Dynamic Multi-Expert Aggregation

Shoubin Yu, Yue Zhang, Ziyang Wang, Jaehong Yoon, Mohit Bansal

机构 * UNC Chapel Hill（北卡罗来纳大学教堂山分校） ； Nanyang Technological University（南洋理工大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments EMNLP 2025 Findings; The first two authors contributed equally; Github link: https://github.com/Yui010206/MEXA

3D Audio-Visual Segmentation

Artem Sokolov, Swapnil Bhosale, Xiatian Zhu

机构 * University of Surrey, UK（Surrey大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted at the NeurIPS 2024 Workshop on Audio Imagination; this version updates the project page link

Information-Theoretic Criteria for Knowledge Distillation in Multimodal Learning

Rongrong Xie, Yizhou Xu, Guido Sanguinetti

机构 * Scuola Internazionale Superiore di Studi Avanzati (SISSA)（国际先进研究高等学院） ； École Polytechnique Fédérale de Lausanne (EPFL)（日内瓦联邦理工学院） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)

Controllable Audio-Visual Viewpoint Generation from 360° Spatial Information

Christian Marinoni, Riccardo Fosco Gramaccioni, Eleonora Grassucci, Danilo Comminiello

机构 * Sapienza University of Rome, Italy（罗马大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Towards fairer public transit: Real-time tensor-based multimodal fare evasion and fraud detection

Peter Wauyo, Dalia Bwiza, Alain Murara, Edwin Mugume, Eric Umuhoza

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)

Comments 10 pages

Multimodal Feedback for Task Guidance in Augmented Reality

Hu Guo, Lily Patel, Rohan Gupt

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)

DM-Codec: Distilling Multimodal Representations for Speech Tokenization

Md Mubtasim Ahasan, Md Fahim, Tasnim Mohiuddin, A K M Mahbubur Rahman, Aman Chadha, Tariq Iqbal, M Ashraful Amin, Md Mofijul Islam, Amin Ahsan Ali

机构 * Center for Computational & Data Sciences, Independent University, Bangladesh（计算与数据科学中心，独立大学，孟加拉国） ； Amazon GenAI（亚马逊生成人工智能） ； Qatar Computing Research Institute（卡塔尔计算研究所） ； University of Virginia（弗吉尼亚大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted at EMNLP 2025

AISHELL6-whisper: A Chinese Mandarin Audio-visual Whisper Speech Dataset with Speech Recognition Baselines

Cancan Li, Fei Su, Juan Liu, Hui Bu, Yulong Wan, Hongbin Suo, Ming Li

机构 * School of Computer Science, Wuhan University（武汉大学计算机学院） ； School of Artificial Intelligence, Wuhan University（武汉大学人工智能学院） ； Beijing AISHELL Technology Co., Ltd.（北京AISHELL科技有限公司） ； AI Center, OPPO（OPPO人工智能中心） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Visual Authority and the Rhetoric of Health Misinformation: A Multimodal Analysis of Social Media Videos

Mohammad Reza Zarei, Barbara Stead-Coyle, Michael Christensen, Sarah Everts, Majid Komeili

机构 * School of Computer Science（计算机科学学院） ； Carleton University（卡尔顿大学） ； Department of Law and Legal Studies（法律与法律研究系） ； School of Journalism and Communication（新闻与传播学院） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Advancing Speech Summarization in Multi-modal LLMs with Reinforcement Learning

Shaoshi Ling, Gang Liu, Guoli Ye, Jinyu Li

机构 * Microsoft CoreAI（微软核心人工智能） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Pay More Attention To Audio: Mitigating Imbalance of Cross-Modal Attention in Large Audio Language Models

Junyu Wang, Ziyang Ma, Zhengding Luo, Tianrui Wang, Meng Ge, Xiaobao Wang, Longbiao Wang

机构 * 1Laboratory of Cognitive Computing ； Application, College of Intelligence ； Computing, Tianjin University, Tianjin, China 2School of Computer Science, Shanghai Jiao Tong University, Shanghai, China 3School of Electrical \& Electronic Engineering, Nanyang Technological University, Singapore 4Huiyan Technology (Tianjin) Co., Ltd, Tianjin, China 纠错

专题命中 音频语音多模态 ：cross-modal(title)；multi-modal(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Submitted to ICASSP 2026

A Multimodal Symphony: Integrating Taste and Sound through Generative AI

Matteo Spanio, Massimiliano Zampini, Antonio Rodà, Franco Pierucci

机构 * Centro di Sonologia Computazionale (CSC)（计算声学中心） ； Department of Information Engineering University of Padova（信息工程系帕多瓦大学） ； Center for Mind/Brain Sciences (CIMeC)（心智/大脑科学中心） ； University of Trento（特伦托大学） ； SoundFood s.r.l.（SoundFood公司） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI、cs.MM、eess.AS

Comments 17 pages, 6 figures (2 + 2 figures with 2 subfigures each)

Journal ref Front. Comput. Sci. 7:1575741 (2025)

Progressive Semantic Residual Quantization for Multimodal-Joint Interest Modeling in Music Recommendation

Shijia Wang, Tianpei Ouyang, Qiang Xiao, Dongjing Wang, Yintao Ren, Songpei Xu, Da Guo, Chuanjiang Luo

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)

JPS: Jailbreak Multimodal Large Language Models with Collaborative Visual Perturbation and Textual Steering

Renmiao Chen, Shiyao Cui, Xuancheng Huang, Chengwei Pan, Victor Shea-Jay Huang, QingLin Zhang, Xuan Ouyang, Zhexin Zhang, Hongning Wang, Minlie Huang

机构 * CoAI group, DCST, Tsinghua University（清华大学DCST学院） ； Beihang University（北航大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Comments 10 pages, 3 tables, 2 figures, to appear in the Proceedings of the 33rd ACM International Conference on Multimedia (MM '25)

Investigating the Invertibility of Multimodal Latent Spaces: Limitations of Optimization-Based Methods

Siwoo Park

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI、eess.AS

Zero-AVSR: Zero-Shot Audio-Visual Speech Recognition with LLMs by Learning Language-Agnostic Speech Representations

Jeong Hun Yeo, Minsu Kim, Chae Won Kim, Stavros Petridis, Yong Man Ro

机构 * KAIST（韩国科学技术院） ； Imperial College London（伦敦帝国理工学院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted at ICCV 2025. Code available at: https://github.com/JeongHun0716/zero-avsr

I see what you mean: Co-Speech Gestures for Reference Resolution in Multimodal Dialogue

Esam Ghaleb, Bulat Khaertdinov, Aslı Özyürek, Raquel Fernández

机构 * Multimodal Language Department, Max Planck Institute for Psycholinguistics（马克斯·普朗克心理语言学研究所多模态语言部门） ； Donders Institute for Brain, Cognition and Behaviour, Radboud University（拉德堡德大学大脑、认知与行为研究所） ； Department of Advanced Computing Sciences, Maastricht University（马斯特里赫特大学高级计算科学系） ； Institute for Logic, Language and Computation, University of Amsterdam（阿姆斯特丹大学逻辑、语言与计算研究所） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted to Findings of ACL 2025

ASR-enhanced Multimodal Representation Learning for Cross-Domain Product Retrieval

Ruixiang Zhao, Jian Jia, Yan Li, Xuehan Bai, Quan Chen, Han Li, Peng Jiang, Xirong Li

机构 * Renmin University of China（中国人民大学） ； Kuaishou Technology（快手科技） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments accepted for publication as a REGULAR paper in the IEEE Transactions on Multimedia

PMF-CEC: Phoneme-augmented Multimodal Fusion for Context-aware ASR Error Correction with Error-specific Selective Decoding

Jiajun He, Tomoki Toda

机构 * Graduate School of Informatics, Nagoya University（信息学研究科，名古屋大学） ； Information Technology Center, Nagoya University（信息科技中心，名古屋大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted by IEEE TASLP 2025

Towards disentangling the contributions of articulation and acoustics in multimodal phoneme recognition

Sean Foley, Hong Nguyen, Jihwan Lee, Sudarsana Reddy Kadiri, Dani Byrd, Louis Goldstein, Shrikanth Narayanan

机构 * University of Southern CaliforniaUSA（美国南加州大学） ； Signal Analysis and Interpretation Laboratory（信号分析与解释实验室） ； Department of Linguistics（语言学系） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)

Audio-Visual Contact Classification for Tree Structures in Agriculture

Ryan Spears, Moonyoung Lee, George Kantor, Oliver Kroemer

机构 * Carnegie Mellon University（卡内基梅隆大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)

Comments 8 pages

Incorporating brain-inspired mechanisms for multimodal learning in artificial intelligence

Xiang He, Dongcheng Zhao, Yang Li, Qingqun Kong, Xin Yang, Yi Zeng

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)

Comments The manuscript is under review and the code is available at https://github.com/Brain-Cog-Lab/IEMF

Audio-Visual Class-Incremental Learning for Fish Feeding intensity Assessment in Aquaculture

Meng Cui, Xianghu Yue, Xinyuan Qian, Jinzheng Zhao, Haohe Liu, Xubo Liu, Daoliang Li, Wenwu Wang

机构 * Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey（视觉、语音和信号处理中心（CVSSP）， Surrey大学） ； College of Intelligence and Computing, Tianjin University（智能与计算学院，天津大学） ； School of Computer and Communication Engineering, University of Science and Technology Beijing（计算机与通信工程学院，北京科技大学） ； National Innovation Center for Digital Fishery, China Agricultural University（数字渔业国家创新中心，中国农业大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)