多模态大模型

Shared Multi-modal Embedding Space for Face-Voice Association

用于人脸-语音关联的共享多模态嵌入空间

Christopher Simic, Korbinian Riedhammer, Tobias Bocklet

专题命中 音频语音多模态 ：multi-modal(title)；分类 cs.CV

AI总结 本文提出了一种基于共享多模态嵌入空间的人脸-语音关联方法，通过自适应角边距损失实现跨语言的高效识别

Comments Ranked 1st in Fame 2026 Challenge, ICASSP

PromptSep: Generative Audio Separation via Multimodal Prompting

Yutong Wen, Ke Chen, Prem Seetharaman, Oriol Nieto, Jiaqi Su, Rithesh Kumar, Minje Kim, Paris Smaragdis, Zeyu Jin, Justin Salamon

机构 * Adobe Research（Adobe研究院） ； University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） ； MIT（麻省理工学院） 纠错

专题命中 音频语音多模态 ：multimodal(title)；分类 eess.AS

Comments Submitted to ICASSP 2026

A Trustworthy Method for Multimodal Emotion Recognition

Junxiao Xue, Xiaozhen Liu, Jie Wang, Xuecheng Wu, Bin Wu

机构 * Big Data Mining and Analytics, xxxxxxx 20xx, x(x): xxx-xxx（大数据挖掘与分析，xxxxx 20xx, x(x): xxx-xxx） 纠错

专题命中 音频语音多模态 ：multimodal(title)；分类 cs.CV

Comments Accepted for publication in Big Data Mining and Analytics (BDMA), 2025

Multimodal Representation Loss Between Timed Text and Audio for Regularized Speech Separation

Tsun-An Hsieh, Heeyoul Choi, Minje Kim

专题命中 音频语音多模态 ：multimodal(title)；分类 eess.AS

Journal ref Interspeech 2024

MiSTR: Multi-Modal iEEG-to-Speech Synthesis with Transformer-Based Prosody Prediction and Neural Phase Reconstruction

Mohammed Salah Al-Radhi, Géza Németh, Branislav Gerazov

专题命中 音频语音多模态 ：multi-modal(title)；分类 eess.AS

Comments 5 pages, 2 figures, 1 table. Accepted for presentation at Interspeech 2025

Sanidha: A Studio Quality Multi-Modal Dataset for Carnatic Music

Venkatakrishnan Vaidyanathapuram Krishnan, Noel Alben, Anish Nair, Nathaniel Condit-Schultz

专题命中 音频语音多模态 ：multi-modal(title)；分类 eess.AS

Comments Accepted to the 25th International Society for Music Information Retrieval Conference (ISMIR 2024)

Journal ref Proceedings of the 25th International Society for Music Information Retrieval Conference, 705-712. San Francisco, California, USA and Online, November 10-14, 2024

Dynamic Inter-Class Confusion-Aware Encoder for Audio-Visual Fusion in Human Activity Recognition

Kaixuan Cong, Yifan Wang, Rongkun Xue, Yuyang Jiang, Yiming Feng, Jing Yang

机构 * School of Automation Science and Engineering, Xi’an Jiaotong University, Xi’an, China（自动化科学与工程学院，西安交通大学，西安，中国） ； School of Software Engineering, Xi’an Jiaotong University, Xi’an, China（软件工程学院，西安交通大学，西安，中国） 纠错

专题命中 音频语音多模态 ：audio-visual(title)；分类 cs.CV

Speak2Sign3D: A Multi-modal Pipeline for English Speech to American Sign Language Animation

Kazi Mahathir Rahman, Naveed Imtiaz Nafis, Md. Farhan Sadik, Mohammad Al Rafi, Mehedi Hasan Shahed

机构 * Department of Computer Science, BRAC University（计算机科学系，布拉克大学） 纠错

专题命中 音频语音多模态 ：multi-modal(title)；分类 cs.CV

Comments 11 pages, 12 figures

Multimodal Integration Challenges in Emotionally Expressive Child Avatars for Training Applications

Pegah Salehi, Sajad Amouei Sheshkal, Vajira Thambawita, Michael A. Riegler, Pål Halvorsen

机构 * SimulaMet 纠错

专题命中 音频语音多模态 ：multimodal(title)；分类 cs.CV

Comments 20 pages, 9 figures, 9 tables

Audio-visual child-adult speaker classification in dyadic interactions

Anfeng Xu, Kevin Huang, Tiantian Feng, Helen Tager-Flusberg, Shrikanth Narayanan

机构 * University of Southern California（南加州大学） ； Boston University（波士顿大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title)；分类 eess.AS

Comments In review for ICASSP 2024, 5 pages

Enhancing Speech Emotion Recognition with Graph-Based Multimodal Fusion and Prosodic Features for the Speech Emotion Recognition in Naturalistic Conditions Challenge at Interspeech 2025

Alef Iury Siqueira Ferreira, Lucas Rafael Gris, Alexandre Ferro Filho, Lucas Ólives, Daniel Ribeiro, Luiz Fernando, Fernanda Lustosa, Rodrigo Tanaka, Frederico Santos de Oliveira, Arlindo Galvão Filho

机构 * Federal University of Goiás（戈亚斯联邦大学） ； Federal University of Rio Grande do Norte（北里奥格兰德联邦大学） ； Aeronautics Institute of Technology（航空技术研究所） ； Federal University of Mato Grosso（马托格罗索联邦大学） 纠错

专题命中 音频语音多模态 ：multimodal(title)；分类 cs.CL

TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorization

Hugo Malard, Michel Olvera, Stephane Lathuiliere, Slim Essid

机构 * LTCI, Télécom Paris, Institut Polytechnique de Paris（LTCI，巴黎电信学院，巴黎高等理工学院） 纠错

专题命中 音频语音多模态 ：audio-visual(title)；分类 eess.AS

Mozualization: Crafting Music and Visual Representation with Multimodal AI

Wanfang Xu, Lixiang Zhao, Haiwen Song, Xinheng Song, Zhaolin Lu, Yu Liu, Min Chen, Eng Gee Lim, Lingyun Yu

机构 * Xi’an Jiaotong-Liverpool University（西安交通大学利物浦大学） ； Beijing Institute of Technology（北京理工大学） 纠错

专题命中 音频语音多模态 ：multimodal(title)；分类 cs.AI

Comments 7 pages, 5 figures, CHI2025

CLAPSep: Leveraging Contrastive Pre-trained Model for Multi-Modal Query-Conditioned Target Sound Extraction

Hao Ma, Zhiyuan Peng, Xu Li, Mingjie Shao, Xixin Wu, Ju Liu

专题命中 音频语音多模态 ：multi-modal(title)；分类 eess.AS

Comments Published in: IEEE/ACM Transactions on Audio, Speech, and Language Processing ( Volume: 32), DOI: 10.1109/TASLP.2024.3497586

Cosh-DiT: Co-Speech Gesture Video Synthesis via Hybrid Audio-Visual Diffusion Transformers

Yasheng Sun, Zhiliang Xu, Hang Zhou, Jiazhi Guan, Quanwei Yang, Kaisiyuan Wang, Borong Liang, Yingying Li, Haocheng Feng, Jingdong Wang, Ziwei Liu, Koike Hideki

专题命中 音频语音多模态 ：audio-visual(title)；分类 cs.CV

Comments Project Page: https://sunyasheng.github.io/projects/COSH-DIT

Uncertainty Modeling in Multimodal Speech Analysis Across the Psychosis Spectrum

Morteza Rohanian, Roya M. Hüppi, Farhad Nooralahzadeh, Noemi Dannecker, Yves Pauli, Werner Surbeck, Iris Sommer, Wolfram Hinzen, Nicolas Langer, Michael Krauthammer, Philipp Homan

专题命中 音频语音多模态 ：multimodal(title)；分类 cs.CL

NeckCare: Preventing Tech Neck using Hearable-based Multimodal Sensing

Bhawana Chhaglani, Alan Seefeldt

专题命中 音频语音多模态 ：multimodal(title)；分类 eess.AS

AVS-Net: Audio-Visual Scale Net for Self-supervised Monocular Metric Depth Estimation

Xiaohu Liu, Sascha Hornauer, Fabien Moutarde, Jialiang Lu

专题命中 音频语音多模态 ：audio-visual(title)；分类 cs.CV

MIMIC: Multimodal Islamophobic Meme Identification and Classification

S M Jishanul Islam, Sahid Hossain Mustakim, Sadia Ahmmed, Md. Faiyaz Abdullah Sayeedi, Swapnil Khandoker, Syed Tasdid Azam Dhrubo, Nahid Hossain

专题命中 音频语音多模态 ：multimodal(title)；分类 cs.CV

Comments Accepted (Poster) - NeurIPS 2024 Workshop MusIML

Multi-modal Adversarial Training for Zero-Shot Voice Cloning

John Janiczek, Dading Chong, Dongyang Dai, Arlo Faria, Chao Wang, Tao Wang, Yuzong Liu

专题命中 音频语音多模态 ：multi-modal(title)；分类 eess.AS

Comments Accepted at INTERSPEECH 2024

Improvement Speaker Similarity for Zero-Shot Any-to-Any Voice Conversion of Whispered and Regular Speech

Anastasia Avdeeva, Aleksei Gusev

专题命中 音频语音多模态 ：any-to-any(title)；分类 eess.AS

Comments Accepted at INTERSPEECH 2024

Creating an Aligned Corpus of Sound and Text: The Multimodal Corpus of Shakespeare and Milton

Manex Agirrezabal

专题命中 音频语音多模态 ：multimodal(title)；分类 cs.CL

FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matching

Chaeyoung Jung, Suyeon Lee, Ji-Hoon Kim, Joon Son Chung

专题命中 音频语音多模态 ：audio-visual(title)；分类 eess.AS

Comments INTERSPEECH 2024

EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars

Nikita Drobyshev, Antoni Bigata Casademunt, Konstantinos Vougioukas, Zoe Landgraf, Stavros Petridis, Maja Pantic

专题命中 音频语音多模态 ：multimodal(title)；分类 cs.CV

Characterizing Multimedia Information Environment through Multi-modal Clustering of YouTube Videos

Niloofar Yousefi, Mainuddin Shaik, Nitin Agarwal

专题命中 音频语音多模态 ：multi-modal(title)；分类 cs.MM

Comments 14 pages, In the 4th International Conference on SMART MULTIMEDIA, 2024

An Exploratory Study of Multimodal Physiological Data in Jazz Improvisation Using Basic Machine Learning Techniques

Yawen Zhang

专题命中 音频语音多模态 ：multimodal(title)；分类 eess.AS

Comments Master's thesis

Enhancing Consistency in Multimodal Dialogue System Using LLM with Dialogue Scenario

Hiroki Onozeki, Zhiyang Qi, Kazuma Akiyama, Ryutaro Asahara, Takumasa Kaneko, Michimasa Inaba

专题命中 音频语音多模态 ：multimodal(title)；分类 cs.CL

Comments This paper is part of the proceedings of the Dialogue Robot Competition 2023

Predicting recovery following stroke: deep learning, multimodal data and feature selection using explainable AI

Adam White, Margarita Saranti, Artur d'Avila Garcez, Thomas M. H. Hope, Cathy J. Price, Howard Bowman

专题命中 音频语音多模态 ：multimodal(title)；分类 cs.AI

Modality Dropout for Multimodal Device Directed Speech Detection using Verbal and Non-Verbal Features

Gautam Krishna, Sameer Dharur, Oggi Rudovic, Pranay Dighe, Saurabh Adya, Ahmed Hussen Abdelaziz, Ahmed H Tewfik

专题命中 音频语音多模态 ：multimodal(title)；分类 eess.AS

Comments 5 pages

CALM: Contrastive Cross-modal Speaking Style Modeling for Expressive Text-to-Speech Synthesis

Yi Meng, Xiang Li, Zhiyong Wu, Tingtian Li, Zixun Sun, Xinyu Xiao, Chi Sun, Hui Zhan, Helen Meng

专题命中 音频语音多模态 ：cross-modal(title)；分类 eess.AS

Comments Accepted by InterSpeech 2022