多模态大模型

EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting

Guanrou Yang, Chen Yang, Qian Chen, Ziyang Ma, Wenxi Chen, Wen Wang, Tianrui Wang, Yifan Yang, Zhikang Niu, Wenrui Liu, Fan Yu, Zhihao Du, Zhifu Gao, ShiLiang Zhang, Xie Chen

机构 * Shanghai Jiao Tong University（上海交通大学） ； Tongyi Speech Lab（通义语音实验室） ； Tianjin University（天津大学） ； Zhejiang University（浙江大学） ； Shanghai Jiao Tong University, Shanghai Innovation Institute（上海交通大学上海创新研究院） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted at ACMMM 2025

Leveraging Audio and Text Modalities in Mental Health: A Study of LLMs Performance

Abdelrahman A. Ali, Aya E. Fouda, Radwa J. Hanafy, Mohammed E. Fouda

机构 * Compumacy for Artificial Intelligence solutions（人工智能解决方案公司） ； Department of Behavioural Health- Saint Elizabeths Hospital（行为健康部门-圣伊丽莎白医院） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

TAIL: Text-Audio Incremental Learning

Yingfei Sun, Xu Gu, Wei Ji, Hanbin Zhao, Yifang Yin, Roger Zimmermann

机构 * National University of Singapore（新加坡国立大学） ； Renmin University of China（中国人民大学） ； Zhejiang University（浙江大学） 纠错

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments 6 figures, 4 tables

Audio-Vision Contrastive Learning for Phonological Class Recognition

Daiqi Liu, Tomás Arias-Vergara, Jana Hutter, Andreas Maier, Paula Andrea Pérez-Toro

机构 * Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany（模式识别实验室，弗里德里希-亚历山大-埃森纳赫大学） ； Smart Imaging Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany（智能成像实验室，弗里德里希-亚历山大-埃森纳赫大学） ； GITA Lab, Universidad de Antioquia UdeA, Medellín, Colombia（GITA实验室，安托法加斯塔大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments conference to TSD 2025

Spotlighting Partially Visible Cinematic Language for Video-to-Audio Generation via Self-distillation

Feizhen Huang, Yu Wu, Yutian Lin, Bo Du

机构 * School of Computer Science, Wuhan University（武汉大学计算机学院） 纠错

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by IJCAI 2025

Double Entendre: Robust Audio-Based AI-Generated Lyrics Detection via Multi-View Fusion

Markus Frohmann, Gabriel Meseguer-Brocal, Markus Schedl, Elena V. Epure

机构 * Deezer Research（DeepZ研究） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted to ACL 2025 Findings

Text2midi: Generating Symbolic Music from Captions

Keshav Bhandari, Abhinaba Roy, Kyra Wang, Geeta Puri, Simon Colton, Dorien Herremans

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 9 pages, 3 figures, Accepted at the 39th AAAI Conference on Artificial Intelligence (AAAI 2025)

Journal ref Proceedings of the 39th AAAI Conference on Artificial Intelligence (AAAI 2025)

A Survey of Automatic Evaluation Methods on Text, Visual and Speech Generations

Tian Lan, Yang-Hao Zhou, Zi-Ao Ma, Fanshu Sun, Rui-Qing Sun, Junyu Luo, Rong-Cheng Tu, Heyan Huang, Chen Xu, Zhijing Wu, Xian-Ling Mao

机构 * Beijing Institute of Technology（北京理工大学） ； Peking University（北京大学） ； Nanyang Technological University（南洋理工大学） 纠错

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

PHRASED: Phrase Dictionary Biasing for Speech Translation

Peidong Wang, Jian Xue, Rui Zhao, Junkun Chen, Aswin Shanmugam Subramanian, Jinyu Li

机构 * Microsoft USA（微软公司） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Teaching Physical Awareness to LLMs through Sounds

Weiguo Wang, Andy Nie, Wenrui Zhou, Yi Kai, Chengchen Hu

机构 * Peking University（北京大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments ICML 2025

TESU-LLM: Training Speech-LLMs Without Speech via Unified Encoder Alignment

Taesoo Kim, Jong Hwan Ko

机构 * KT Corporation（KT公司） ； Department of Electrical and Computer Engineering（电气与计算机工程系） ； Sungkyunkwan University（成均馆大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

DeepFake Doctor: Diagnosing and Treating Audio-Video Fake Detection

Marcel Klemt, Carlotta Segna, Anna Rohrbach

机构 * TU Darmstadt（图宾根大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Towards Reliable Large Audio Language Model

Ziyang Ma, Xiquan Li, Yakun Song, Wenxi Chen, Chenpeng Du, Jian Wu, Yuanzhe Chen, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xie Chen

机构 * X-LANCE Lab, School of Computer Science, MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University（X-LANCE实验室，计算机科学学院，人工智能国家重点实验室，上海交通大学） ； ByteDance（字节跳动） 纠错

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments ACL 2025 Findings

Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translation

Yuhao Zhang, Xiangnan Ma, Kaiqi Kou, Peizhuo Liu, Weiqiao Shan, Benyou Wang, Tong Xiao, Yuxin Huang, Zhengtao Yu, Jingbo Zhu

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted to ACL 2025 Findings

Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering

Gang Li, Jizhong Liu, Heinrich Dinkel, Yadong Niu, Junbo Zhang, Jian Luan

机构 * Xiaomi Corporation（小米公司） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

BLAB: Brutally Long Audio Bench

Orevaoghene Ahia, Martijn Bartelds, Kabir Ahuja, Hila Gonen, Valentin Hofmann, Siddhant Arora, Shuyue Stella Li, Vishal Puttagunta, Mofetoluwa Adeyemi, Charishma Buchireddy, Ben Walls, Noah Bennett, Shinji Watanabe, Noah A. Smith, Yulia Tsvetkov, Sachin Kumar

机构 * University of Washington（华盛顿大学） ； Stanford University（斯坦福大学） ； The Ohio State University（俄亥俄州立大学） ； Carnegie Mellon University（卡内基梅隆大学） ； University of Waterloo（多伦多大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Tell What You Hear From What You See -- Video to Audio Generation Through Text

Xiulong Liu, Kun Su, Eli Shlizerman

机构 * University of Washington（华盛顿大学） 纠错

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments NeurIPS 2024. Project page: https://dragonliu1995.github.io/VATT-home

Qwen2.5-Omni Technical Report

Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, Junyang Lin

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、eess.AS

Video-to-Audio Generation with Hidden Alignment

Manjie Xu, Chenxing Li, Xinyi Tu, Yong Ren, Rilin Chen, Yu Gu, Wei Liang, Dong Yu

机构 * Tencent AI Lab（腾讯AI实验室） ； Beijing Institute of Technology（北京理工大学） ； University of California, Berkeley（加州大学伯克利分校） ； Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所） 纠错

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments https://sites.google.com/view/vta-ldm

DiVISe: Direct Visual-Input Speech Synthesis Preserving Speaker Characteristics And Intelligibility

Yifan Liu, Yu Fang, Zhouhan Lin

机构 * Shanghai Jiao Tong University（上海交通大学） ； ShanghaiTech University（上海科技大学） 纠错

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments to be published in NAACL 25

Bridging the Data Provenance Gap Across Text, Speech and Video

Shayne Longpre, Nikhil Singh, Manuel Cherep, Kushagra Tiwary, Joanna Materzynska, William Brannon, Robert Mahari, Naana Obeng-Marnu, Manan Dey, Mohammed Hamdy, Nayan Saxena, Ahmad Mustafa Anis, Emad A. Alghamdi, Vu Minh Chien, Da Yin, Kun Qian, Yizhi Li, Minnie Liang, An Dinh, Shrestha Mohanty, Deividas Mataciunas, Tobin South, Jianguo Zhang, Ariel N. Lee, Campbell S. Lund, Christopher Klamm, Damien Sileo, Diganta Misra, Enrico Shippole, Kevin Klyman, Lester JV Miranda, Niklas Muennighoff, Seonghyeon Ye, Seungone Kim, Vipul Gupta, Vivek Sharma, Xuhui Zhou, Caiming Xiong, Luis Villa, Stella Biderman, Alex Pentland, Sara Hooker, Jad Kabbara

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、cs.MM

Comments ICLR 2025. 10 pages, 5 figures (main paper)

Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction

Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, Peng Liu, Ruihang Miao, Wang You, Xi Chen, Xuerui Yang, Yechang Huang, Yuxiang Zhang, Zheng Gong, Zixin Zhang, Hongyu Zhou, Jianjian Sun, Brian Li, Chengting Feng, Changyi Wan, Hanpeng Hu, Jianchang Wu, Jiangjie Zhen, Ranchen Ming, Song Yuan, Xuelin Zhang, Yu Zhou, Bingxin Li, Buyun Ma, Hongyuan Wang, Kang An, Wei Ji, Wen Li, Xuan Wen, Xiangwen Kong, Yuankai Ma, Yuanwei Liang, Yun Mou, Bahtiyar Ahmidi, Bin Wang, Bo Li, Changxin Miao, Chen Xu, Chenrun Wang, Dapeng Shi, Deshan Sun, Dingyuan Hu, Dula Sai, Enle Liu, Guanzhe Huang, Gulin Yan, Heng Wang, Haonan Jia, Haoyang Zhang, Jiahao Gong, Junjing Guo, Jiashuai Liu, Jiahong Liu, Jie Feng, Jie Wu, Jiaoren Wu, Jie Yang, Jinguo Wang, Jingyang Zhang, Junzhe Lin, Kaixiang Li, Lei Xia, Li Zhou, Liang Zhao, Longlong Gu, Mei Chen, Menglin Wu, Ming Li, Mingxiao Li, Mingliang Li, Mingyao Liang, Na Wang, Nie Hao, Qiling Wu, Qinyuan Tan, Ran Sun, Shuai Shuai, Shaoliang Pang, Shiliang Yang, Shuli Gao, Shanshan Yuan, Siqi Liu, Shihong Deng, Shilei Jiang, Sitong Liu, Tiancheng Cao, Tianyu Wang, Wenjin Deng, Wuxun Xie, Weipeng Ming, Wenqing He, Wen Sun, Xin Han, Xin Huang, Xiaomin Deng, Xiaojia Liu, Xin Wu, Xu Zhao, Yanan Wei, Yanbo Yu, Yang Cao, Yangguang Li, Yangzhen Ma, Yanming Xu, Yaoyu Wang, Yaqiang Shi, Yilei Wang, Yizhuang Zhou, Yinmin Zhong, Yang Zhang, Yaoben Wei, Yu Luo, Yuanwei Lu, Yuhe Yin, Yuchu Luo, Yuanhao Ding, Yuting Yan, Yaqi Dai, Yuxiang Yang, Zhe Xie, Zheng Ge, Zheng Sun, Zhewei Huang, Zhichao Chang, Zhisheng Guan, Zidong Yang, Zili Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, Jiansheng Chen, Jing Li, Shuchang Zhou, Xiangyu Zhang, Xinhao Zhang, Yibo Zhu

机构 * StepFun（阶跃星辰） 纠错

专题命中 音频语音多模态 ：multi-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

The Sound of Water: Inferring Physical Properties from Pouring Liquids

Piyush Bagad, Makarand Tapaswi, Cees G. M. Snoek, Andrew Zisserman

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Project page at https://bpiyush.github.io/pouring-water-website. Short version accepted to ICASSP 2025

Read, Watch and Scream! Sound Generation from Text and Video

Yujin Jeong, Yunji Kim, Sanghyuk Chun, Jiyoung Lee

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments AAAI2025, Project page: https://naver-ai.github.io/rewas

PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis

Yifan Xie, Tao Feng, Xin Zhang, Xiangyang Luo, Zixuan Guo, Weijiang Yu, Heng Chang, Fei Ma, Fei Richard Yu

专题命中 音频语音多模态 ：cross-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments 9 pages, accepted by AAAI 2025

Separate Anything You Describe

Xubo Liu, Qiuqiang Kong, Yan Zhao, Haohe Liu, Yi Yuan, Yuzhuo Liu, Rui Xia, Yuxuan Wang, Mark D. Plumbley, Wenwu Wang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments Code, benchmark and pre-trained models: AGI/AudioSep" target="_blank" rel="noopener">https://github.com/Audio-AGI/AudioSep

GaussianSpeech: Audio-Driven Gaussian Avatars

Shivangi Aneja, Artem Sevastopolsky, Tobias Kirschstein, Justus Thies, Angela Dai, Matthias Nießner

专题命中 音频语音多模态 ：audio-visual(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Paper Video: https://youtu.be/2VqYoFlYcwQ Project Page: https://shivangi-aneja.github.io/projects/gaussianspeech

SALMONN-omni: A Codec-free LLM for Full-duplex Speech Understanding and Generation

Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, Chao Zhang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Technical report

Generative AI for Music and Audio

Hao-Wen Dong

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments PhD Dissertation

Understanding Audiovisual Deepfake Detection: Techniques, Challenges, Human Factors and Perceptual Insights

Ammarah Hashmi, Sahibzada Adil Shahzad, Chia-Wen Lin, Yu Tsao, Hsin-Min Wang

专题命中 音频语音多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM