多模态大模型

TalkSketch: Multimodal Generative AI for Real-time Sketch Ideation with Speech

Weiyan Shi, Sunaya Upadhyay, Geraldine Quek, Kenny Tsu Wei Choo

机构 * Singapore University of Technology and Design（新加坡科技设计大学） ； Carnegie Mellon University（卡内基梅隆大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.MM

Comments Accepted at AAAI 2026 Workshop on Creative AI for Live Interactive Performances (CLIP). To be published in Springer CCIS series

Enhancing XR Auditory Realism via Multimodal Scene-Aware Acoustic Rendering

Tianyu Xu, Jihan Li, Penghe Zu, Pranav Sahay, Maruchi Kim, Jack Obeng-Marnu, Farley Miller, Xun Qian, Katrina Passarella, Mahitha Rachumalla, Rajeev Nongpiur, D. Shin

机构 * Google（谷歌） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV

Journal ref Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology (UIST '25), Article 17, 1-16, 2025

TMDC: A Two-Stage Modality Denoising and Complementation Framework for Multimodal Sentiment Analysis with Missing and Noisy Modalities

Yan Zhuang, Minhao Liu, Yanru Zhang, Jiawen Deng, Fuji Ren

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.MM

Comments Accepted by AAAI 2026

A Comprehensive Survey on Multi-modal Conversational Emotion Recognition with Deep Learning

Yuntao Shou, Tao Meng, Wei Ai, Fangze Fu, Nan Yin, Keqin Li

机构 * Central South University of Forestry and Technology（中部林业科技大学） ； Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学） ； State University of New York（纽约州立大学） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.AI

Comments 36 pages, 10 figures

MCAD: Multimodal Context-Aware Audio Description Generation For Soccer

Lipisha Chaudhary, Trisha Mittal, Subhadra Gopalakrishnan, Ifeoma Nwogu, Jaclyn Pytlarz

机构 * University at Buffalo, SUNY（布法罗大学，SUNY） ； Dolby Laboratories Inc.（杜比实验室公司） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.MM

Two-stage Audio-Visual Target Speaker Extraction System for Real-Time Processing On Edge Device

Zixuan Li, Xueliang Zhang, Lei Miao, Zhipeng Yan, Ying Sun, Chong Zhu

机构 * College of Computer Science, Inner Mongolia University, China（内蒙古大学计算机科学学院） ； Lenovo, China（联想公司） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 eess.AS

Mixed Signals: Understanding Model Disagreement in Multimodal Empathy Detection

Maya Srikanth, Run Chen, Julia Hirschberg

机构 * Columbia University（哥伦比亚大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

Comments To appear in Findings of IJCNLP-AACL 2025

Aligning Brain Signals with Multimodal Speech and Vision Embeddings

Kateryna Shapovalenko, Quentin Auster

机构 * Carnegie Mellon University（卡内基梅隆大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI

Towards Resource-Efficient Multimodal Intelligence: Learned Routing among Specialized Expert Models

Mayank Saini, Arit Kumar Bishwas

机构 * PwC US（普华永道美国） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

Comments 15 pages, 4 figures

Socially Aware Music Recommendation: A Multi-Modal Graph Neural Networks for Collaborative Music Consumption and Community-Based Engagement

Kajwan Ziaoddini

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.MM

Shared Latent Representation for Joint Text-to-Audio-Visual Synthesis

Dogucan Yaman, Seymanur Akti, Fevziye Irem Eyiokur, Alexander Waibel

机构 * Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院） ； Carnegie Mellon University（卡内基梅隆大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV

Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarks

Xu Zheng, Zihao Dongfang, Lutao Jiang, Boyuan Zheng, Yulong Guo, Zhenquan Zhang, Giuliano Albanese, Runyi Yang, Mengjiao Ma, Zixin Zhang, Chenfei Liao, Dingcheng Zhen, Yuanhuiyi Lyu, Yuqian Fu, Bin Ren, Linfeng Zhang, Danda Pani Paudel, Nicu Sebe, Luc Van Gool, Xuming Hu

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV

Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy?

Yiwen Guan, Viet Anh Trinh, Vivek Voleti, Jacob Whitehill

专题命中 音频语音多模态 ：multi-modal(title)；分类 cs.CV、cs.CL、cs.MM

Journal ref IEEE International Conference on Multimedia and Expo (ICME), Nantes, France, 2025, pp. 1-6

Abjad AI at NADI 2025: CATT-Whisper: Multimodal Diacritic Restoration Using Text and Speech Representations

Ahmad Ghannam, Naif Alharthi, Faris Alasmary, Kholood Al Tabash, Shouq Sadah, Lahouari Ghouti

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Serving

Haoran Qiu, Anish Biswas, Zihan Zhao, Jayashree Mohan, Alind Khare, Esha Choukse, Íñigo Goiri, Zeyu Zhang, Haiying Shen, Chetan Bansal, Ramachandran Ramjee, Rodrigo Fonseca

机构 * Microsoft Azure Research（微软Azure研究） ； Microsoft Research India（微软印度研究） ； University of Virginia（弗吉尼亚大学） ； Microsoft M365 Research（微软M365研究） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI

Comments Published at ACM SoCC 2025; 14 pages, 20 figures

Audio-Visual Speech Enhancement for Spatial Audio - Spatial-VisualVoice and the MAVE Database

Danielle Yaffe, Ferdinand Campe, Prachi Sharma, Dorothea Kolossa, Boaz Rafaely

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 eess.AS

Leveraging LLMs for Context-Aware Implicit Textual and Multimodal Hate Speech Detection

Joshua Wolfe Brook, Ilia Markov

机构 * Computational Linguistics \& Text Mining Lab Vrije Universiteit Amsterdam 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

Comments 8 pages, 9 figures, submitted to LREC 2026

Towards Multimodal Query-Based Spatial Audio Source Extraction

Chenxin Yu, Hao Ma, Xu Li, Xiao-Lei Zhang, Mingjie Shao, Chi Zhang, Xuelong Li

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 eess.AS

Comments Submitted to ICASSP 2026

Adaptive vector steering: A training-free, layer-wise intervention for hallucination mitigation in large audio and multimodal models

Tsung-En Lin, Kuan-Yi Lee, Hung-Yi Lee

机构 * National Taiwan University（国立台湾大学） ； ASUS Open Cloud Infrastructure Software Center（ASUS开放云基础设施软件中心） 纠错

专题命中 音频语音多模态 ：multimodal(title)；multi-modal(abstract)；分类 eess.AS

Comments Note: This preprint is a version of the paper submitted to ICASSP 2026. The author list here includes contributors who provided additional supervision and guidance. The official ICASSP submission may differ slightly in author composition

A Fast and Lightweight Model for Causal Audio-Visual Speech Separation

Wendi Sang, Kai Li, Runxuan Yang, Jianqiang Huang, Xiaolin Hu

机构 * School of Computer Technology and Application（计算机技术与应用学院） ； Intelligent Computing and Application Laboratory of Qinghai Province（青海省智能计算与应用实验室） ； Department of Computer Science and Technology（计算机科学与技术系） ； Institute for AI（人工智能研究院） ； Tsinghua Laboratory of Brain and Intelligence (THBI)（清华大学脑与智能实验室） ； IDG/McGovern Institute for Brain Research（IDG/麦克戈维脑研究学院） ； Tsinghua University（清华大学） ； Chinese Institute for Brain Research (CIBR)（中国脑科学研究院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 eess.AS

Comments Accepted by ECAI 2025

Exploring Student Choice and the Use of Multimodal Generative AI in Programming Learning

Xinying Hou, Ruiwei Xiao, Runlong Ye, Michael Liut, John Stamper

机构 * University of Michigan（密歇根大学） ； Carnegie Mellon University（卡内基梅隆大学） ； University of Toronto（多伦多大学） ； University of Toronto Mississauga（多伦多大学滑铁卢分校） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI

Comments 7 pages, accepted to SIGCSE2026

Leveraging Self-Supervised Audio-Visual Pretrained Models to Improve Vocoded Speech Intelligibility in Cochlear Implant Simulation

Richard Lee Lai, Jen-Cheng Hou, I-Chun Chern, Kuo-Hsuan Hung, Yi-Ting Chen, Mandar Gogate, Tughrul Arslan, Amir Hussain, Yu Tsao

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 eess.AS

Leveraging Mamba with Full-Face Vision for Audio-Visual Speech Enhancement

Rong Chao, Wenze Ren, You-Jin Li, Kuo-Hsuan Hung, Sung-Feng Huang, Szu-Wei Fu, Wen-Huang Cheng, Yu Tsao

机构 * Academia Sinica（台湾“中央研究院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 eess.AS

Comments Accepted to Interspeech 2025 Workshop

Advancing Audio-Visual Navigation Through Multi-Agent Collaboration in 3D Environments

Hailong Zhang, Yinfeng Yu, Liejun Wang, Fuchun Sun, Wendong Zheng

机构 * Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center（新疆多模态智能处理与信息安全工程技术研究中心） ； School of Computer Science and Technology, Xinjiang University（新疆大学计算机科学与技术学院） ； Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系） ； School of Electrical Engineering and Automation, Tianjin University of Technology（天津理工大学电气工程与自动化学院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.AI

Comments Main paper (15 pages). Accepted for publication by ICONIP( International Conference on Neural Information Processing) 2025

Thinking with Sound: Audio Chain-of-Thought Enables Multimodal Reasoning in Large Audio-Language Models

Zhen Xiong, Yujun Cai, Zhecheng Li, Junsong Yuan, Yiwei Wang

机构 * University of Southern California（南加州大学） ； University of Queensland（昆士兰大学） ； University of California, San Diego（加州大学圣地亚哥分校） ； University of Buffalo（布法罗大学） ； University of California, Merced（加州大学默塞德分校） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

Longitudinal and Multimodal Recording System to Capture Real-World Patient-Clinician Conversations for AI and Encounter Research: Protocol

Misk Al Zahidy, Kerly Guevara Maldonado, Luis Vilatuna Andrango, Ana Cristina Proano, Ana Gabriela Claros, Maria Lizarazo Jimenez, David Toro-Tobon, Victor M. Montori, Oscar J. Ponce-Ponte, Juan P. Brito

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

Comments 23 pages, 2 figures, 2 tables

Real-Time System for Audio-Visual Target Speech Enhancement

T. Aleksandra Ma, Sile Yin, Li-Chia Yang, Shuo Zhang

机构 * Bose Corporation（博世公司） ； Georgia Institute of Technology（佐治亚理工学院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 eess.AS

Comments Accepted into WASPAA 2025 demo session

PoolingVQ: A VQVAE Variant for Reducing Audio Redundancy and Boosting Multi-Modal Fusion in Music Emotion Analysis

Dinghao Zou, Yicheng Gong, Xiaokang Li, Xin Cao, Sunbowen Lee

机构 * Wuhan University of Science and Technology（武汉科技大学） 纠错

专题命中 音频语音多模态 ：multi-modal(title)；multimodal(abstract)；分类 eess.AS

The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audio

Renhao Wang, Haoran Geng, Tingle Li, Feishi Wang, Gopala Anumanchipalli, Trevor Darrell, Boyi Li, Pieter Abbeel, Jitendra Malik, Alexei A. Efros

机构 * University of California, Berkeley（加州大学伯克利分校） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV

Comments Conference on Robot Learning 2025

Interpreting the Role of Visemes in Audio-Visual Speech Recognition

Aristeidis Papadopoulos, Naomi Harte

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 eess.AS

Comments Accepted into Automatic Speech Recognition and Understanding- ASRU 2025