多模态大模型

Speak, Edit, Repeat: High-Fidelity Voice Editing and Zero-Shot TTS with Cross-Attentive Mamba

Baher Mohammad, Magauiya Zhussip, Stamatios Lefkimmiatis

机构 * MTS AI（MTS人工智能公司） ； ITMO University（ITMO大学） 纠错

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

VoiceAssistant-Eval: Benchmarking AI Assistants across Listening, Speaking, and Viewing

Ke Wang, Houxing Ren, Zimu Lu, Mingjie Zhan, Hongsheng Li

机构 * CUHK MMLab（香港中文大学多媒体实验室） ； SenseTime Research（商汤科技研究院） ； CPII under InnoHK（创新香港下的CPII） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

VAInpaint: Zero-Shot Video-Audio inpainting framework with LLMs-driven Module

Kam Man Wu, Zeyue Tian, Liya Ji, Qifeng Chen

机构 * The Hong Kong University of Science and Technology（香港科学与技术大学） 纠错

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

MERaLiON-SpeechEncoder: Towards a Speech Foundation Model for Singapore and Beyond

Muhammad Huzaifah, Geyu Lin, Tianchi Liu, Hardik B. Sailor, Kye Min Tan, Tarun K. Vangani, Qiongqiong Wang, Jeremy H. M. Wong, Jinyang Wu, Nancy F. Chen, Ai Ti Aw

机构 * MERaLiON Team（MERaLiON团队） ； Institute for Infocomm Research (I 2 R), A*STAR, Singapore（信息通信研究所（I2R），A*STAR，新加坡） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech

Siyi Zhou, Yiquan Zhou, Yi He, Xun Zhou, Jinchao Wang, Wei Deng, Jingchen Shu

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CL、cs.AI、eess.AS

Improving Child Speech Recognition and Reading Mistake Detection by Using Prompts

Lingyun Gao, Cristian Tejedor-Garcia, Catia Cucchiarini, Helmer Strik

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments This paper is accepted to Interspeech 2025. This publication is part of the project Responsible AI for Voice Diagnostics (RAIVD) with file number NGF.1607.22.013 of the research programme NGF AiNed Fellowship Grants which is financed by the Dutch Research Council (NWO)

Journal ref https://www.isca-archive.org/interspeech_2025/gao25c_interspeech.html

FlowDubber: Movie Dubbing with LLM-based Semantic-aware Learning and Flow Matching based Voice Enhancing

Gaoxiang Cong, Liang Li, Jiadong Pan, Zhedong Zhang, Amin Beheshti, Anton van den Hengel, Yuankai Qi, Qingming Huang

机构 * Institute of Computing Technology, Chinese Academy of Sciences \& University of Chinese Academy of Sciences Beijing China ； Institute of Computing Technology, Chinese Academy of Sciences Beijing China ； Hangzhou Dianzi University Hangzhou China ； Macquarie University Sydney Australia ； University of Adelaide Adelaide Australia ； University of Chinese Academy of Sciences Beijing China ； Institute of Computing Technology, Chinese Academy of Sciences \& University of Chinese Academy of Sciences ； Institute of Computing Technology, Chinese Academy of Sciences ； Hangzhou Dianzi University ； Macquarie University ； University of Adelaide ； University of Chinese Academy of Sciences 纠错

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Sentiment Reasoning for Healthcare

Khai-Nguyen Nguyen, Khai Le-Duc, Bach Phan Tat, Duy Le, Long Vo-Dang, Truong-Son Hy

机构 * College of William and Mary（威廉与玛丽学院） ； University of Toronto（多伦多大学） ； University Health Network（大学健康网络） ； KU Leuven（鲁汶大学） ； Bucknell University（巴克内尔大学） ； University of Cincinnati（辛辛那提大学） ； University of Alabama at Birmingham（伯明翰大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments ACL 2025 Industry Track (Oral)

EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting

Guanrou Yang, Chen Yang, Qian Chen, Ziyang Ma, Wenxi Chen, Wen Wang, Tianrui Wang, Yifan Yang, Zhikang Niu, Wenrui Liu, Fan Yu, Zhihao Du, Zhifu Gao, ShiLiang Zhang, Xie Chen

机构 * Shanghai Jiao Tong University（上海交通大学） ； Tongyi Speech Lab（通义语音实验室） ； Tianjin University（天津大学） ； Zhejiang University（浙江大学） ； Shanghai Jiao Tong University, Shanghai Innovation Institute（上海交通大学上海创新研究院） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted at ACMMM 2025

Leveraging Audio and Text Modalities in Mental Health: A Study of LLMs Performance

Abdelrahman A. Ali, Aya E. Fouda, Radwa J. Hanafy, Mohammed E. Fouda

机构 * Compumacy for Artificial Intelligence solutions（人工智能解决方案公司） ； Department of Behavioural Health- Saint Elizabeths Hospital（行为健康部门-圣伊丽莎白医院） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

TAIL: Text-Audio Incremental Learning

Yingfei Sun, Xu Gu, Wei Ji, Hanbin Zhao, Yifang Yin, Roger Zimmermann

机构 * National University of Singapore（新加坡国立大学） ； Renmin University of China（中国人民大学） ； Zhejiang University（浙江大学） 纠错

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments 6 figures, 4 tables

Audio-Vision Contrastive Learning for Phonological Class Recognition

Daiqi Liu, Tomás Arias-Vergara, Jana Hutter, Andreas Maier, Paula Andrea Pérez-Toro

机构 * Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany（模式识别实验室，弗里德里希-亚历山大-埃森纳赫大学） ； Smart Imaging Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany（智能成像实验室，弗里德里希-亚历山大-埃森纳赫大学） ； GITA Lab, Universidad de Antioquia UdeA, Medellín, Colombia（GITA实验室，安托法加斯塔大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments conference to TSD 2025

Spotlighting Partially Visible Cinematic Language for Video-to-Audio Generation via Self-distillation

Feizhen Huang, Yu Wu, Yutian Lin, Bo Du

机构 * School of Computer Science, Wuhan University（武汉大学计算机学院） 纠错

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by IJCAI 2025

Double Entendre: Robust Audio-Based AI-Generated Lyrics Detection via Multi-View Fusion

Markus Frohmann, Gabriel Meseguer-Brocal, Markus Schedl, Elena V. Epure

机构 * Deezer Research（DeepZ研究） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted to ACL 2025 Findings

Text2midi: Generating Symbolic Music from Captions

Keshav Bhandari, Abhinaba Roy, Kyra Wang, Geeta Puri, Simon Colton, Dorien Herremans

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 9 pages, 3 figures, Accepted at the 39th AAAI Conference on Artificial Intelligence (AAAI 2025)

Journal ref Proceedings of the 39th AAAI Conference on Artificial Intelligence (AAAI 2025)

A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generations

Tian Lan, Yang-Hao Zhou, Zi-Ao Ma, Fanshu Sun, Rui-Qing Sun, Junyu Luo, Rong-Cheng Tu, Heyan Huang, Chen Xu, Zhijing Wu, Xian-Ling Mao

机构 * Beijing Institute of Technology（北京理工大学） ； Peking University（北京大学） ； Nanyang Technological University（南洋理工大学） 纠错

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

PHRASED: Phrase Dictionary Biasing for Speech Translation

Peidong Wang, Jian Xue, Rui Zhao, Junkun Chen, Aswin Shanmugam Subramanian, Jinyu Li

机构 * Microsoft USA（微软公司） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Teaching Physical Awareness to LLMs through Sounds

Weiguo Wang, Andy Nie, Wenrui Zhou, Yi Kai, Chengchen Hu

机构 * Peking University（北京大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments ICML 2025

TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignment

Taesoo Kim, Jong Hwan Ko

机构 * KT Corporation（KT公司） ； Department of Electrical and Computer Engineering（电气与计算机工程系） ； Sungkyunkwan University（成均馆大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

DeepFake Doctor: Diagnosing and Treating Audio-Video Fake Detection

Marcel Klemt, Carlotta Segna, Anna Rohrbach

机构 * TU Darmstadt（图宾根大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Towards Reliable Large Audio Language Model

Ziyang Ma, Xiquan Li, Yakun Song, Wenxi Chen, Chenpeng Du, Jian Wu, Yuanzhe Chen, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xie Chen

机构 * X-LANCE Lab, School of Computer Science, MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University（X-LANCE实验室，计算机科学学院，人工智能国家重点实验室，上海交通大学） ； ByteDance（字节跳动） 纠错

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments ACL 2025 Findings

Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translation

Yuhao Zhang, Xiangnan Ma, Kaiqi Kou, Peizhuo Liu, Weiqiao Shan, Benyou Wang, Tong Xiao, Yuxin Huang, Zhengtao Yu, Jingbo Zhu

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted to ACL 2025 Findings

Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering

Gang Li, Jizhong Liu, Heinrich Dinkel, Yadong Niu, Junbo Zhang, Jian Luan

机构 * Xiaomi Corporation（小米公司） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

BLAB: Brutally Long Audio Bench

Orevaoghene Ahia, Martijn Bartelds, Kabir Ahuja, Hila Gonen, Valentin Hofmann, Siddhant Arora, Shuyue Stella Li, Vishal Puttagunta, Mofetoluwa Adeyemi, Charishma Buchireddy, Ben Walls, Noah Bennett, Shinji Watanabe, Noah A. Smith, Yulia Tsvetkov, Sachin Kumar

机构 * University of Washington（华盛顿大学） ； Stanford University（斯坦福大学） ； The Ohio State University（俄亥俄州立大学） ； Carnegie Mellon University（卡内基梅隆大学） ； University of Waterloo（多伦多大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Tell What You Hear From What You See -- Video to Audio Generation Through Text

Xiulong Liu, Kun Su, Eli Shlizerman

机构 * University of Washington（华盛顿大学） 纠错

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments NeurIPS 2024. Project page: https://dragonliu1995.github.io/VATT-home

Qwen2.5-Omni Technical Report

Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, Junyang Lin

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、eess.AS

Video-to-Audio Generation with Hidden Alignment

Manjie Xu, Chenxing Li, Xinyi Tu, Yong Ren, Rilin Chen, Yu Gu, Wei Liang, Dong Yu

机构 * Tencent AI Lab（腾讯AI实验室） ； Beijing Institute of Technology（北京理工大学） ； University of California, Berkeley（加州大学伯克利分校） ； Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所） 纠错

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments https://sites.google.com/view/vta-ldm

DiVISe: Direct Visual-Input Speech Synthesis Preserving Speaker Characteristics And Intelligibility

Yifan Liu, Yu Fang, Zhouhan Lin

机构 * Shanghai Jiao Tong University（上海交通大学） ； ShanghaiTech University（上海科技大学） 纠错

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments to be published in NAACL 25

Bridging the Data Provenance Gap Across Text, Speech and Video

Shayne Longpre, Nikhil Singh, Manuel Cherep, Kushagra Tiwary, Joanna Materzynska, William Brannon, Robert Mahari, Naana Obeng-Marnu, Manan Dey, Mohammed Hamdy, Nayan Saxena, Ahmad Mustafa Anis, Emad A. Alghamdi, Vu Minh Chien, Da Yin, Kun Qian, Yizhi Li, Minnie Liang, An Dinh, Shrestha Mohanty, Deividas Mataciunas, Tobin South, Jianguo Zhang, Ariel N. Lee, Campbell S. Lund, Christopher Klamm, Damien Sileo, Diganta Misra, Enrico Shippole, Kevin Klyman, Lester JV Miranda, Niklas Muennighoff, Seonghyeon Ye, Seungone Kim, Vipul Gupta, Vivek Sharma, Xuhui Zhou, Caiming Xiong, Luis Villa, Stella Biderman, Alex Pentland, Sara Hooker, Jad Kabbara

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、cs.MM

Comments ICLR 2025. 10 pages, 5 figures (main paper)

Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction

Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, Peng Liu, Ruihang Miao, Wang You, Xi Chen, Xuerui Yang, Yechang Huang, Yuxiang Zhang, Zheng Gong, Zixin Zhang, Hongyu Zhou, Jianjian Sun, Brian Li, Chengting Feng, Changyi Wan, Hanpeng Hu, Jianchang Wu, Jiangjie Zhen, Ranchen Ming, Song Yuan, Xuelin Zhang, Yu Zhou, Bingxin Li, Buyun Ma, Hongyuan Wang, Kang An, Wei Ji, Wen Li, Xuan Wen, Xiangwen Kong, Yuankai Ma, Yuanwei Liang, Yun Mou, Bahtiyar Ahmidi, Bin Wang, Bo Li, Changxin Miao, Chen Xu, Chenrun Wang, Dapeng Shi, Deshan Sun, Dingyuan Hu, Dula Sai, Enle Liu, Guanzhe Huang, Gulin Yan, Heng Wang, Haonan Jia, Haoyang Zhang, Jiahao Gong, Junjing Guo, Jiashuai Liu, Jiahong Liu, Jie Feng, Jie Wu, Jiaoren Wu, Jie Yang, Jinguo Wang, Jingyang Zhang, Junzhe Lin, Kaixiang Li, Lei Xia, Li Zhou, Liang Zhao, Longlong Gu, Mei Chen, Menglin Wu, Ming Li, Mingxiao Li, Mingliang Li, Mingyao Liang, Na Wang, Nie Hao, Qiling Wu, Qinyuan Tan, Ran Sun, Shuai Shuai, Shaoliang Pang, Shiliang Yang, Shuli Gao, Shanshan Yuan, Siqi Liu, Shihong Deng, Shilei Jiang, Sitong Liu, Tiancheng Cao, Tianyu Wang, Wenjin Deng, Wuxun Xie, Weipeng Ming, Wenqing He, Wen Sun, Xin Han, Xin Huang, Xiaomin Deng, Xiaojia Liu, Xin Wu, Xu Zhao, Yanan Wei, Yanbo Yu, Yang Cao, Yangguang Li, Yangzhen Ma, Yanming Xu, Yaoyu Wang, Yaqiang Shi, Yilei Wang, Yizhuang Zhou, Yinmin Zhong, Yang Zhang, Yaoben Wei, Yu Luo, Yuanwei Lu, Yuhe Yin, Yuchu Luo, Yuanhao Ding, Yuting Yan, Yaqi Dai, Yuxiang Yang, Zhe Xie, Zheng Ge, Zheng Sun, Zhewei Huang, Zhichao Chang, Zhisheng Guan, Zidong Yang, Zili Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, Jiansheng Chen, Jing Li, Shuchang Zhou, Xiangyu Zhang, Xinhao Zhang, Yibo Zhu

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、cs.AI、eess.AS