多模态大模型

Audio-centric Video Understanding Benchmark without Text Shortcut

Yudong Yang, Jimin Zhuang, Guangzhi Sun, Changli Tang, Yixuan Li, Peihan Li, Yifan Jiang, Wei Li, Zejun Ma, Chao Zhang

机构 * Tsinghua University（清华大学） ； University of Cambridge（剑桥大学） ； ByteDance（字节跳动） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI

Comments Accepted for publication in the Proceedings of EMNLP 2025 (Main Conference)

Learning from Silence and Noise for Visual Sound Source Localization

Xavier Juanola, Giovana Morais, Magdalena Fuentes, Gloria Haro

机构 * Intelligent Multimodal Vision Analysis Universitat Pompeu Fabra Barcelona, Spain（智能多模态视觉分析Universitat Pompeu Fabra Barcelona） ； MARL-IDM New York University, New York, USA（MARL-IDM纽约大学） 纠错

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM

Comments 10 pages, 2 figures, 4 tables + Supplementary Material

When Audio and Text Disagree: Revealing Text Bias in Large Audio-Language Models

Cheng Wang, Gelei Deng, Xianglin Yang, Han Qiu, Tianwei Zhang

机构 * National University of Singapore（国立新加坡大学） ； Nanyang Technological University（南洋理工大学） ； Tsinghua University（清华大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CL、cs.AI

Comments Accepted by EMNLP 2025 Main

Wearable Music2Emotion : Assessing Emotions Induced by AI-Generated Music through Portable EEG-fNIRS Fusion

Sha Zhao, Song Yi, Yangxuan Zhou, Jiadong Pan, Jiquan Wang, Jie Xia, Shijian Li, Shurong Dong, Gang Pan

机构 * Zhejiang University（浙江大学） ； Hangzhou RongNao Technology Co., Ltd（杭州融脑科技有限公司） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.AI、eess.AS

Comments Accepted by ACM MM 2025

Can Sound Replace Vision in LLaVA With Token Substitution?

Ali Vosoughi, Jing Bi, Pinxin Liu, Yunlong Tang, Chenliang Xu

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.MM、eess.AS

Comments Project page: https://ali-vosoughi.github.io/SoundCLIP/

Task Arithmetic for Language Expansion in Speech Translation

Yao-Fei Cheng, Hayato Futami, Yosuke Kashiwagi, Emiru Tsunoo, Wen Shen Teo, Siddhant Arora, Shinji Watanabe

机构 * University of Washington（华盛顿大学） ； Sony Group Corporation（索尼集团） ； University of Electro-Communications（电通大学） ； Carnegie Mellon University（卡内基梅隆大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；multimodal foundation model(abstract)；分类 cs.CL、cs.AI

DualDub: Video-to-Soundtrack Generation via Joint Speech and Background Audio Synthesis

Wenjie Tian, Xinfa Zhu, Haohe Liu, Zhixian Zhao, Zihao Chen, Chaofan Ding, Xinhan Di, Junjie Zheng, Lei Xie

机构 * Northwestern Polytechnical University（西北工业大学） ； Centre for Vision Speech and Signal Processing, University of Surrey（视觉语音与信号处理中心，萨里大学） ； AI Lab, Giant Network（巨人网络人工智能实验室） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.MM、eess.AS

Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks

Chien-yu Huang, Wei-Chih Chen, Shu-wen Yang, Andy T. Liu, Chen-An Li, Yu-Xiang Lin, Wei-Cheng Tseng, Anuj Diwan, Yi-Jen Shih, Jiatong Shi, William Chen, Chih-Kai Yang, Wenze Ren, Xuanjun Chen, Chi-Yuan Hsiao, Puyuan Peng, Shih-Heng Wang, Chun-Yi Kuan, Ke-Han Lu, Kai-Wei Chang, Fabian Ritter-Gutierrez, Kuan-Po Huang, Siddhant Arora, You-Kuan Lin, Ming To Chuang, Eunjung Yeo, Kalvin Chang, Chung-Ming Chien, Kwanghee Choi, Jun-You Wang, Cheng-Hsiu Hsieh, Yi-Cheng Lin, Chee-En Yu, I-Hsiang Chiu, Heitor R. Guimarães, Jionghao Han, Tzu-Quan Lin, Tzu-Yuan Lin, Homu Chang, Ting-Wu Chang, Chun Wei Chen, Shou-Jen Chen, Yu-Hua Chen, Hsi-Chun Cheng, Kunal Dhawan, Jia-Lin Fang, Shi-Xin Fang, Kuan-Yu Fang Chiang, Chi An Fu, Hsien-Fu Hsiao, Ching Yu Hsu, Shao-Syuan Huang, Lee Chen Wei, Hsi-Che Lin, Hsuan-Hao Lin, Hsuan-Ting Lin, Jian-Ren Lin, Ting-Chun Liu, Li-Chun Lu, Tsung-Min Pai, Ankita Pasad, Shih-Yun Shan Kuan, Suwon Shon, Yuxun Tang, Yun-Shao Tsai, Jui-Chiang Wei, Tzu-Chieh Wei, Chengxi Wu, Dien-Ruei Wu, Chao-Han Huck Yang, Chieh-Chi Yang, Jia Qi Yip, Shao-Xiang Yuan, Vahid Noroozi, Zhehuai Chen, Haibin Wu, Karen Livescu, David Harwath, Shinji Watanabe, Hung-yi Lee

专题命中 音频语音多模态 ：multimodal(abstract)；multimodal foundation model(abstract)；分类 cs.CL、eess.AS

Comments ICLR 2025

Tell me Habibi, is it Real or Fake?

Kartik Kuckreja, Parul Gupta, Injy Hamed, Thamar Solorio, Muhammad Haris Khan, Abhinav Dhall

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI

Comments 9 pages, 2 figures, 12 tables

Learning to Highlight Audio by Watching Movies

Chao Huang, Ruohan Gao, J. M. F. Tsang, Jan Kurcius, Cagdas Bilen, Chenliang Xu, Anurag Kumar, Sanjeel Parekh

机构 * University of Rochester（罗切斯特大学） ； University of Maryland, College Park（马里兰大学学院公园分校） ； Meta Reality Labs Research（Meta现实实验室） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments CVPR 2025. Project page: https://wikichao.github.io/VisAH/

SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer

Young-Hu Park, Rae-Hong Park, Hyung-Min Park

专题命中 音频语音多模态 ：multi-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Journal ref Neurocomputing, Volume 639, 28 July 2025, 130289

Detecting Children with Autism Spectrum Disorder based on Script-Centric Behavior Understanding with Emotional Enhancement

Wenxing Liu, Yueran Pan, Dong Zhang, Hongzhu Deng, Xiaobing Zou, Ming Li

机构 * School of Computer Science, Wuhan University（武汉大学计算机学院） ； Digital Innovation Research Center, Duke Kunshan University（杜克昆山大学数字创新研究中心） ； School of Electronics and Information Technology, Sun Yat-sen University（中山大学电子与信息学院） ； Child Development and Behavior Center, Third Affiliated Hospital of Sun Yat-sen University（中山大学第三附属医院儿童发展与行为中心） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI

Comments 15 pages, 12 figures, sumbitted to IEEE transactions on affective computing

Qieemo: Speech Is All You Need in the Emotion Recognition in Conversations

Jinming Chen, Jingyi Fang, Yuanzhong Zheng, Yaoxuan Wang, Haojun Fei

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.AI、eess.AS

DeepSound-V1: Start to Think Step-by-Step in the Audio Generation from Videos

Yunming Liang, Zihao Chen, Chaofan Ding, Xinhan Di

专题命中 音频语音多模态 ：multi-modal(abstract)；MLLM(abstract)；分类 cs.CV、eess.AS

Comments 11 pages, 6 figures

Seeing Speech and Sound: Distinguishing and Locating Audios in Visual Scenes

Hyeonggon Ryu, Seongyu Kim, Joon Son Chung, Arda Senocak

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments CVPR 2025

Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics

Chen Liu, Liying Yang, Peike Li, Dadong Wang, Lincheng Li, Xin Yu

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted by CVPR2025

Seeing Sound: Assembling Sounds from Visuals for Audio-to-Image Generation

Darius Petermann, Mahdi M. Kalayeh

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Linguistics-Vision Monotonic Consistent Network for Sign Language Production

Xu Wang, Shengeng Tang, Peipei Song, Shuo Wang, Dan Guo, Richang Hong

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted by ICASSP 2025

A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation

Louis Airale, Dominique Vaufreydaz, Xavier Alameda-Pineda

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Large-scale unsupervised audio pre-training for video-to-speech synthesis

Triantafyllos Kefalas, Yannis Panagakis, Maja Pantic

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Corrected typos. This work has been submitted to the IEEE for possible publication

Speech Recognition Rescoring with Large Speech-Text Foundation Models

Prashanth Gurunath Shivakumar, Jari Kolehmainen, Aditya Gourav, Yi Gu, Ankur Gandhe, Ariya Rastrow, Ivan Bulyko

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation

Se Jin Park, Chae Won Kim, Hyeongseop Rha, Minsu Kim, Joanna Hong, Jeong Hun Yeo, Yong Man Ro

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI

Comments Accepted to ACL 2024 (Oral)

Generative Expressive Conversational Speech Synthesis

Rui Liu, Yifan Hu, Yi Ren, Xiang Yin, Haizhou Li

专题命中 音频语音多模态 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CL、eess.AS

Comments 14 pages, 6 figures, 8 tables. Accepted by ACM MM 2024

Face-voice Association in Multilingual Environments (FAME) Challenge 2024 Evaluation Plan

Muhammad Saad Saeed, Shah Nawaz, Muhammad Salman Tahir, Rohan Kumar Das, Muhammad Zaigham Zaheer, Marta Moscati, Markus Schedl, Muhammad Haris Khan, Karthik Nandakumar, Muhammad Haroon Yousaf

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments ACM Multimedia Conference - Grand Challenge

An End-to-End Speech Summarization Using Large Language Model

Hengchao Shang, Zongyao Li, Jiaxin Guo, Shaojun Li, Zhiqiang Rao, Yuanchang Luo, Daimeng Wei, Hao Yang

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Comments InterSpeech 2024

WikiMuTe: A web-sourced dataset of semantic descriptions for music audio

Benno Weck, Holger Kirchhoff, Peter Grosche, Xavier Serra

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Comments Submitted to 30th International Conference on MultiMedia Modeling (MMM2024). This preprint has not undergone peer review or any post-submission improvements or corrections

Journal ref The Version of Record of this contribution is published in MultiMedia Modeling. MMM 2024. Lecture Notes in Computer Science, vol 14565. Springer, Cham

SALMONN: Towards Generic Hearing Abilities for Large Language Models

Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Chao Zhang

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Speech Emotion Recognition with Distilled Prosodic and Linguistic Affect Representations

Debaditya Shome, Ali Etemad

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI

Comments Accepted at ICASSP 2024

Novel-View Acoustic Synthesis

Changan Chen, Alexander Richard, Roman Shapovalov, Vamsi Krishna Ithapu, Natalia Neverova, Kristen Grauman, Andrea Vedaldi

专题命中 音频语音多模态 ：multi-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted at CVPR 2023. Project page: https://vision.cs.utexas.edu/projects/nvas

Separating Invisible Sounds Toward Universal Audiovisual Scene-Aware Sound Separation

Yiyang Su, Ali Vosoughi, Shijian Deng, Yapeng Tian, Chenliang Xu

专题命中 音频语音多模态 ：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted at ICCV 2023 - AV4D, 4 figures, 3 tables