多模态大模型 - arXivDaily 专题

2505.13455 2025-06-02 eess.AS cs.AI 62%

Spatiotemporal Emotional Synchrony in Dyadic Interactions: The Role of Speech Conditions in Facial and Vocal Affective Alignment

Von Ralph Dane Marquez Herbuela, Yukie Nagai

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、eess.AS

2502.09284 2025-06-02 cs.CL cs.AI 62%

SparQLe: Speech Queries to Text Translation Through LLMs

Amirbek Djanibekov, Hanan Aldarmaki

机构 * Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、cs.AI

2505.23308 2025-05-30 eess.AS cs.AI eess.IV 62%

Spoken question answering for visual queries

Nimrod Shabtay, Zvi Kons, Avihu Dekel, Hagai Aronowitz, Ron Hoory, Assaf Arbelle

机构 * IBM Research（IBM研究院）； Tel-Aviv University（特拉维夫大学）

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.AI、eess.AS

Comments Accepted for Interspeech 2025 (with additional results)

2505.21148 2025-05-28 cs.CL cs.SD eess.AS 62%

Assessment of L2 Oral Proficiency using Speech Large Language Models

Rao Ma, Mengjie Qian, Siyuan Tang, Stefano Bannò, Kate M. Knill, Mark J. F. Gales

机构 * University of Cambridge（剑桥大学）； ALTA Institute（ALTA研究院）

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments submitted to Interspeech

2505.19978 2025-05-27 cs.CL cs.SD eess.AS 62%

DeepDialogue: A Multi-Turn Emotionally-Rich Spoken Dialogue Dataset

Alkis Koudounas, Moreno La Quatra, Elena Baralis

机构 * Politecnico di Torino（托里尼理工大学）； Kore University of Enna（恩纳科雷大学）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Currently under review. See the official website: https://salt-research.github.io/DeepDialogue

2409.01995 2025-05-27 eess.AS cs.AI cs.SD 62%

vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoders

Yiwei Guo, Zhihan Li, Junjie Li, Chenpeng Du, Hankun Wang, Shuai Wang, Xie Chen, Kai Yu

机构 * X-LANCE Lab（X-LANCE实验室）； MoE Key lab of Artificial Intelligence（人工智能关键实验室）； School of Computer Science（计算机科学学院）； Shanghai Jiao Tong University（上海交通大学）； Jiangsu Key Lab of Language Computing（江苏语言计算关键实验室）； Shenzhen Research Institute of Big Data（深圳大数据研究 institute）

专题命中音频语音多模态：any-to-any(abstract)；分类 cs.AI、eess.AS

Comments 5 pages, 3 figures, 2 tables. Demo page: https://cantabile-kwok.github.io/vec2wav2/

2505.17426 2025-05-26 cs.SD cs.AI eess.AS 62%

UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic information

Rui Wang, Qianguo Sun, Tianrong Chen, Zhiyun Zeng, Junlong Wu, Jiaxing Zhang

机构 * International Digital Economy Academy（国际数字经济学院）； IDEA&Emdoor Collaborative Laboratory（IDEA与Emdoor协同实验室）； Emdoor Information Co., Ltd（Emdoor信息有限公司）； Shenzhen Yijiayiban Information Technology Co., Ltd（深圳艺佳iban信息技术有限公司）

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.AI、eess.AS

2505.15773 2025-05-22 eess.AS cs.CL cs.SD 62%

ToxicTone: A Mandarin Audio Dataset Annotated for Toxicity and Toxic Utterance Tonality

Yu-Xiang Luo, Yi-Cheng Lin, Ming-To Chuang, Jia-Hung Chen, I-Ning Tsai, Pei Xing Kiew, Yueh-Hsuan Huang, Chien-Feng Liu, Yu-Chen Chen, Bo-Han Feng, Wenze Ren, Hung-yi Lee

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted by INTERSPEECH 2025. 5 pages

2505.14887 2025-05-22 cs.CL cs.SD eess.AS 62%

In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties

Nathan Roll, Calbert Graham, Yuka Tatsumi, Kim Tien Nguyen, Meghan Sumner, Dan Jurafsky

机构 * Stanford University（斯坦福大学）； University of Cambridge（剑桥大学）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments 15 pages; 3 figures

2505.12654 2025-05-21 cs.CL cs.AI 62%

Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals

Yuxin Lin, Yinglin Zheng, Ming Zeng, Wangzheng Shi

机构 * School of Informatics, Xiamen University（厦门大学信息学院）

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、cs.AI

Comments Accepected by ACL 2025

2305.03568 2025-05-12 cs.SD cs.LG cs.MM eess.AS 62%

A vector quantized masked autoencoder for audiovisual speech emotion recognition

Samir Sadok, Simon Leglaive, Renaud Séguier

机构 * CentraleSupélec（中央理工学院）； IETR（信息与电信研究实验室）； UMR CNRS 6164（法国国家科学研究中心6164号研究单位）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.MM、eess.AS

Comments 13 pages, 6 figures, https://samsad35.github.io/VQ-MAE-AudioVisual/

2406.04321 2025-05-08 cs.CV cs.LG cs.MM cs.SD 62%

VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling

Zeyue Tian, Zhaoyang Liu, Ruibin Yuan, Jiahao Pan, Qifeng Liu, Xu Tan, Qifeng Chen, Wei Xue, Yike Guo

机构 * Hong Kong University of Science and Technology（香港科技大学）； Microsoft Research Asia（微软亚洲研究院）

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM

Comments The code and datasets are available at https://github.com/ZeyueT/VidMuse/

2504.20447 2025-04-30 cs.SD cs.AI eess.AS 62%

APG-MOS: Auditory Perception Guided-MOS Predictor for Synthetic Speech

Zhicheng Lian, Lizhi Wang, Hua Huang

机构 * Beijing Normal University（北京师范大学）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、eess.AS

2504.15093 2025-04-22 cs.CL cs.AI cs.LG 62%

Rethinking the Potential of Multimodality in Collaborative Problem Solving Diagnosis with Large Language Models

K. Wong, B. Wu, S. Bulathwela, M. Cukurova

机构 * UCL Knowledge Lab, Institute of Education, University College London, UK（伦敦大学学院教育研究所知识实验室）； AI Centre, Dept. of Computer Science, University College London, UK（伦敦大学学院人工智能中心）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI

Comments Accepted for 26th International Conference on Artificial Intelligence in Education (AIED 2025), 22 - 26 July 2025, Palermo, Italy. 17 pages, 1 figure

2501.03181 2025-04-17 cs.SD cs.AI eess.AS 62%

FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles

Tian-Hao Zhang, Jiawei Zhang, Jun Wang, Xinyuan Qian, Xu-Cheng Yin

机构 * Tencent AI Lab（腾讯AI实验室）

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.AI、eess.AS

Comments Accepted by AAAI 2025

2504.09516 2025-04-15 cs.SD cs.CV eess.AS 62%

FSSUAVL: A Discriminative Framework using Vision Models for Federated Self-Supervised Audio and Image Understanding

Yasar Abbas Ur Rehman, Kin Wai Lau, Yuyang Xie, Ma Lan, JiaJun Shen

机构 * TCL AI Lab（TCL人工智能实验室）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、eess.AS

Comments 8 pages

2312.09736 2025-04-15 cs.CL cs.SD eess.AS 62%

HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue

Sunjae Yoon, Dahyun Kim, Eunseop Yoon, Hee Suk Yoon, Junyeong Kim, Chnag D. Yoo

机构 * Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院（KAIST））； Chung-Ang University（中央大学）

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments EMNLP 2023, 14 pages, 13 figures

2504.07229 2025-04-11 cs.CL eess.AS eess.SP 62%

Visual-Aware Speech Recognition for Noisy Scenarios

Lakshmipathi Balaji, Karan Singla

机构 * IIIT-Hyderabad（印度国际信息技术学院海得拉巴分校）； Whissle Inc.（惠斯勒公司）

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CL、eess.AS

2504.04427 2025-04-08 cs.CV cs.AI cs.HC 62%

FluentLip: A Phonemes-Based Two-stage Approach for Audio-Driven Lip Synthesis with Optical Flow Consistency

Shiyan Liu, Rui Qu, Yan Jin

机构 * Huazhong University of Science and Technology（华中科技大学）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.AI

2504.02386 2025-04-04 cs.CV eess.AS 62%

VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models

Kim Sung-Bin, Jeongsoo Choi, Puyuan Peng, Joon Son Chung, Tae-Hyun Oh, David Harwath

机构 * POSTECH（浦项科技大学）； KAIST（韩国科学技术院）； The University of Texas at Austin（德克萨斯大学奥斯汀分校）

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments https://voicecraft-dub.github.io/

2504.00750 2025-04-02 cs.SD cs.LG cs.MM eess.AS 62%

$C^2$AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction

Wenxuan Wu, Xueyuan Chen, Shuai Wang, Jiadong Wang, Lingwei Meng, Xixin Wu, Helen Meng, Haizhou Li

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.MM、eess.AS

Comments Accepted by IEEE Journal of Selected Topics in Signal Processing (JSTSP)

2503.23395 2025-04-01 cs.SD cs.AI eess.AS 62%

Scaling Auditory Cognition via Test-Time Compute in Audio Language Models

Ting Dang, Yan Gao, Hong Jia

机构 * School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）； University of Cambridge（剑桥大学）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、eess.AS

2407.02264 2025-04-01 cs.CV cs.SD eess.AS 62%

SOAF: Scene Occlusion-aware Neural Acoustic Field

Huiyu Gao, Jiahao Ma, David Ahmedt-Aristizabal, Chuong Nguyen, Miaomiao Liu

机构 * Australian National University（澳大利亚国立大学）； CSIRO Data61

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、eess.AS

2503.22275 2025-03-31 eess.AS cs.AI cs.SD 62%

Make Some Noise: Towards LLM audio reasoning and generation using sound tokens

Shivam Mehta, Nebojsa Jojic, Hannes Gamper

机构 * KTH Royal Institute of Technology（瑞典皇家理工学院）； Microsoft Research（微软研究院）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、eess.AS

Comments 5 pages, 2 figures, Accepted at ICASSP 2025

2405.08295 2025-03-26 cs.CL cs.SD eess.AS 62%

SpeechVerse: A Large-scale Generalizable Audio Language Model

Nilaksh Das, Saket Dingliwal, Srikanth Ronanki, Rohit Paturi, Zhaocheng Huang, Prashant Mathur, Jie Yuan, Dhanush Bekal, Xing Niu, Sai Muralidhar Jayanthi, Xilai Li, Karel Mundnich, Monica Sunkara, Sravan Bodapati, Sundararajan Srinivasan, Kyu J Han, Katrin Kirchhoff

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Single Column, 13 page

2409.18042 2025-03-21 cs.CV cs.CL 62%

EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions

Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang, Yi Zhu, Yihan Zeng, Kuo Yang, Dingdong Wang, Kun Xiang, Haoyuan Li, Haoli Bai, Jianhua Han, Xiaohui Li, Weike Jin, Nian Xie, Yu Zhang, James T. Kwok, Hengshuang Zhao, Xiaodan Liang, Dit-Yan Yeung, Xiao Chen, Zhenguo Li, Wei Zhang, Qun Liu, Jun Yao, Lanqing Hong, Lu Hou, Hang Xu

机构 * Hong Kong University of Science and Technology（香港科技大学）； The University of Hong Kong（香港大学）； Huawei Noah’s Ark Lab（华为诺亚方舟实验室）； The Chinese University of Hong Kong（香港中文大学）； Sun Yat-sen University（中山大学）； Southern University of Science and Technology（南方科技大学）

专题命中音频语音多模态：omni-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted by CVPR 2025. Project Page: https://emova-ollm.github.io/

2503.15338 2025-03-20 eess.AS cs.CL cs.SD 62%

Solla: Towards a Speech-Oriented LLM That Hears Acoustic Context

Junyi Ao, Dekun Chen, Xiaohai Tian, Wenjie Feng, Jun Zhang, Lu Lu, Yuxuan Wang, Haizhou Li, Zhizheng Wu

机构 * School of Data Science, Shenzhen Research Institute of Big Data, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）数据科学学院、深圳大数据研究院）； Bytedance（字节跳动）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

2503.14185 2025-03-19 cs.CL cs.SD eess.AS 62%

AdaST: Dynamically Adapting Encoder States in the Decoder for End-to-End Speech-to-Text Translation

Wuwei Huang, Dexin Wang, Deyi Xiong

机构 * College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CL、eess.AS

Comments ACL 2021 Findings

2406.08920 2025-03-18 cs.SD cs.AI eess.AS 62%

AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis

Swapnil Bhosale, Haosen Yang, Diptesh Kanojia, Jiankang Deng, Xiatian Zhu

机构 * University of Surrey（萨里大学）； Imperial College London（帝国理工学院）

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.AI、eess.AS

Comments Accepted to NeurIPS 2024

2501.17202 2025-03-13 cs.SD cs.CL eess.AS 62%

Audio Large Language Models Can Be Descriptive Speech Quality Evaluators

Chen Chen, Yuchen Hu, Siyin Wang, Helin Wang, Zhehuai Chen, Chao Zhang, Chao-Han Huck Yang, Eng Siong Chng

机构 * Nanyang Technological University（南洋理工大学）； NVIDIA（英伟达）； Tsinghua University（清华大学）； Johns Hopkins University（约翰斯·霍普金斯大学）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments ICLR 2025

1. 音频语音多模态 4597 篇

Spatiotemporal Emotional Synchrony in Dyadic Interactions: The Role of Speech Conditions in Facial and Vocal Affective Alignment

SparQLe: Speech Queries to Text Translation Through LLMs

Spoken question answering for visual queries

Assessment of L2 Oral Proficiency using Speech Large Language Models

DeepDialogue: A Multi-Turn Emotionally-Rich Spoken Dialogue Dataset

vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoders

UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic information

ToxicTone: A Mandarin Audio Dataset Annotated for Toxicity and Toxic Utterance Tonality

In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties

Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals

A vector quantized masked autoencoder for audiovisual speech emotion recognition

VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling

APG-MOS: Auditory Perception Guided-MOS Predictor for Synthetic Speech

Rethinking the Potential of Multimodality in Collaborative Problem Solving Diagnosis with Large Language Models

FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles

FSSUAVL: A Discriminative Framework using Vision Models for Federated Self-Supervised Audio and Image Understanding

HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue

Visual-Aware Speech Recognition for Noisy Scenarios

FluentLip: A Phonemes-Based Two-stage Approach for Audio-Driven Lip Synthesis with Optical Flow Consistency

VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models

$C^2$AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction

Scaling Auditory Cognition via Test-Time Compute in Audio Language Models

SOAF: Scene Occlusion-aware Neural Acoustic Field

Make Some Noise: Towards LLM audio reasoning and generation using sound tokens

SpeechVerse: A Large-scale Generalizable Audio Language Model

EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions

Solla: Towards a Speech-Oriented LLM That Hears Acoustic Context

AdaST: Dynamically Adapting Encoder States in the Decoder for End-to-End Speech-to-Text Translation

AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis

Audio Large Language Models Can Be Descriptive Speech Quality Evaluators