多模态大模型 - arXivDaily 专题

2402.03710 2025-06-12 eess.AS cs.CL cs.SD 62%

Listen, Chat, and Remix: Text-Guided Soundscape Remixing for Enhanced Auditory Experience

Xilin Jiang, Cong Han, Yinghao Aaron Li, Nima Mesgarani

机构 * Department of Electrical Engineering, Columbia University（电气工程系，哥伦比亚大学）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted by IEEE Journal of Selected Topics in Signal Processing (JSTSP)

2506.08634 2025-06-11 cs.HC cs.AI cs.CV 62%

MOSAIC-F: A Framework for Enhancing Students' Oral Presentation Skills through Personalized Feedback

Alvaro Becerra, Daniel Andres, Pablo Villegas, Roberto Daza, Ruth Cobos

机构 * GHIA Group, School of Engineering, Universidad Autónoma de Madrid, Spain（GHIA小组，工程学院，马德里自治大学，西班牙）； BiDA-Lab Group, School of Engineering, Universidad Autónoma de Madrid, Spain（BiDA实验室小组，工程学院，马德里自治大学，西班牙）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Accepted in LASI Spain 25: Learning Analytics Summer Institute Spain 2025

2506.08279 2025-06-11 cs.CV cs.AI cs.LG 62%

Seeing Voices: Generating A-Roll Video from Audio with Mirage

Aditi Sundararaman, Amogh Adishesha, Andrew Jaegle, Dan Bigioi, Hyoung-Kyu Song, Jon Kyl, Justin Mao, Kevin Lan, Mojtaba Komeili, ShahRukh Athar, Sheila Babayan, Stanislau Beliasau, William Buchwalter

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Technical report website: mirage.app/research/seeing-voices, product website: mirage.app

2506.07572 2025-06-10 cs.CV cs.CL 62%

Learning Speaker-Invariant Visual Features for Lipreading

Yu Li, Feng Xue, Shujie Li, Jinrui Zhang, Shuang Yang, Dan Guo, Richang Hong

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CV、cs.CL

2506.06537 2025-06-10 cs.CV cs.SD eess.AS 62%

Bridging Audio and Vision: Zero-Shot Audiovisual Segmentation by Connecting Pretrained Models

Seung-jae Lee, Paul Hongsuck Seo

机构 * Department of Computer Science and Engineering（计算机科学与工程系）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、eess.AS

Comments Accepted on INTERSPEECH2025

2506.03214 2025-06-05 q-bio.NC cs.AI cs.CL 62%

A Pre-trained Framework for Multilingual Brain Decoding Using Non-invasive Recordings

Yi Guo, Yihang Dong, Michael Kwok-Po Ng, Shuqiang Wang

机构 * Shenzhen Institute of Advanced Technology（深圳先进技术研究院）； Chinese Academy of Sciences（中国科学院）； University of Chinese Academy of Sciences（中国科学院大学）； Hong Kong Baptist University（香港 Baptist大学）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI

2303.11607 2025-06-05 cs.CL cs.SD eess.AS 62%

Transformers in Speech Processing: A Survey

Siddique Latif, Aun Zaidi, Heriberto Cuayahuitl, Fahad Shamshad, Moazzam Shoukat, Muhammad Usama, Junaid Qadir

机构 * Queensland University of Technology（昆士兰理工大学）； Information Technology University（信息科技大学）； University of Lincoln（林肯大学）； Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）； National University of Computer and Emerging Sciences（国家计算机与新兴科学大学）； Qatar University（卡塔尔大学）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted in Computer Science Review 2025

2506.00722 2025-06-03 cs.CL cs.SD eess.AS 62%

Chain-of-Thought Training for Open E2E Spoken Dialogue Systems

Siddhant Arora, Jinchuan Tian, Hayato Futami, Jee-weon Jung, Jiatong Shi, Yosuke Kashiwagi, Emiru Tsunoo, Shinji Watanabe

机构 * Carnegie Mellon University（卡内基梅隆大学）； Sony Group Corporation（索尼集团）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted at INTERSPEECH 2025

2505.13455 2025-06-02 eess.AS cs.AI 62%

Spatiotemporal Emotional Synchrony in Dyadic Interactions: The Role of Speech Conditions in Facial and Vocal Affective Alignment

Von Ralph Dane Marquez Herbuela, Yukie Nagai

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、eess.AS

2502.09284 2025-06-02 cs.CL cs.AI 62%

SparQLe: Speech Queries to Text Translation Through LLMs

Amirbek Djanibekov, Hanan Aldarmaki

机构 * Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、cs.AI

2505.23308 2025-05-30 eess.AS cs.AI eess.IV 62%

Spoken question answering for visual queries

Nimrod Shabtay, Zvi Kons, Avihu Dekel, Hagai Aronowitz, Ron Hoory, Assaf Arbelle

机构 * IBM Research（IBM研究院）； Tel-Aviv University（特拉维夫大学）

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.AI、eess.AS

Comments Accepted for Interspeech 2025 (with additional results)

2505.21148 2025-05-28 cs.CL cs.SD eess.AS 62%

Assessment of L2 Oral Proficiency using Speech Large Language Models

Rao Ma, Mengjie Qian, Siyuan Tang, Stefano Bannò, Kate M. Knill, Mark J. F. Gales

机构 * University of Cambridge（剑桥大学）； ALTA Institute（ALTA研究院）

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments submitted to Interspeech

2505.19978 2025-05-27 cs.CL cs.SD eess.AS 62%

DeepDialogue: A Multi-Turn Emotionally-Rich Spoken Dialogue Dataset

Alkis Koudounas, Moreno La Quatra, Elena Baralis

机构 * Politecnico di Torino（托里尼理工大学）； Kore University of Enna（恩纳科雷大学）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Currently under review. See the official website: https://salt-research.github.io/DeepDialogue

2409.01995 2025-05-27 eess.AS cs.AI cs.SD 62%

vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoders

Yiwei Guo, Zhihan Li, Junjie Li, Chenpeng Du, Hankun Wang, Shuai Wang, Xie Chen, Kai Yu

机构 * X-LANCE Lab（X-LANCE实验室）； MoE Key lab of Artificial Intelligence（人工智能关键实验室）； School of Computer Science（计算机科学学院）； Shanghai Jiao Tong University（上海交通大学）； Jiangsu Key Lab of Language Computing（江苏语言计算关键实验室）； Shenzhen Research Institute of Big Data（深圳大数据研究 institute）

专题命中音频语音多模态：any-to-any(abstract)；分类 cs.AI、eess.AS

Comments 5 pages, 3 figures, 2 tables. Demo page: https://cantabile-kwok.github.io/vec2wav2/

2505.17426 2025-05-26 cs.SD cs.AI eess.AS 62%

UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic information

Rui Wang, Qianguo Sun, Tianrong Chen, Zhiyun Zeng, Junlong Wu, Jiaxing Zhang

机构 * International Digital Economy Academy（国际数字经济学院）； IDEA&Emdoor Collaborative Laboratory（IDEA与Emdoor协同实验室）； Emdoor Information Co., Ltd（Emdoor信息有限公司）； Shenzhen Yijiayiban Information Technology Co., Ltd（深圳艺佳iban信息技术有限公司）

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.AI、eess.AS

2505.15773 2025-05-22 eess.AS cs.CL cs.SD 62%

ToxicTone: A Mandarin Audio Dataset Annotated for Toxicity and Toxic Utterance Tonality

Yu-Xiang Luo, Yi-Cheng Lin, Ming-To Chuang, Jia-Hung Chen, I-Ning Tsai, Pei Xing Kiew, Yueh-Hsuan Huang, Chien-Feng Liu, Yu-Chen Chen, Bo-Han Feng, Wenze Ren, Hung-yi Lee

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted by INTERSPEECH 2025. 5 pages

2505.14887 2025-05-22 cs.CL cs.SD eess.AS 62%

In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties

Nathan Roll, Calbert Graham, Yuka Tatsumi, Kim Tien Nguyen, Meghan Sumner, Dan Jurafsky

机构 * Stanford University（斯坦福大学）； University of Cambridge（剑桥大学）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments 15 pages; 3 figures

2505.12654 2025-05-21 cs.CL cs.AI 62%

Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals

Yuxin Lin, Yinglin Zheng, Ming Zeng, Wangzheng Shi

机构 * School of Informatics, Xiamen University（厦门大学信息学院）

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、cs.AI

Comments Accepected by ACL 2025

2305.03568 2025-05-12 cs.SD cs.LG cs.MM eess.AS 62%

A vector quantized masked autoencoder for audiovisual speech emotion recognition

Samir Sadok, Simon Leglaive, Renaud Séguier

机构 * CentraleSupélec（中央理工学院）； IETR（信息与电信研究实验室）； UMR CNRS 6164（法国国家科学研究中心6164号研究单位）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.MM、eess.AS

Comments 13 pages, 6 figures, https://samsad35.github.io/VQ-MAE-AudioVisual/

2406.04321 2025-05-08 cs.CV cs.LG cs.MM cs.SD 62%

VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling

Zeyue Tian, Zhaoyang Liu, Ruibin Yuan, Jiahao Pan, Qifeng Liu, Xu Tan, Qifeng Chen, Wei Xue, Yike Guo

机构 * Hong Kong University of Science and Technology（香港科技大学）； Microsoft Research Asia（微软亚洲研究院）

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、cs.MM

Comments The code and datasets are available at https://github.com/ZeyueT/VidMuse/

2504.20447 2025-04-30 cs.SD cs.AI eess.AS 62%

APG-MOS: Auditory Perception Guided-MOS Predictor for Synthetic Speech

Zhicheng Lian, Lizhi Wang, Hua Huang

机构 * Beijing Normal University（北京师范大学）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、eess.AS

2504.15093 2025-04-22 cs.CL cs.AI cs.LG 62%

Rethinking the Potential of Multimodality in Collaborative Problem Solving Diagnosis with Large Language Models

K. Wong, B. Wu, S. Bulathwela, M. Cukurova

机构 * UCL Knowledge Lab, Institute of Education, University College London, UK（伦敦大学学院教育研究所知识实验室）； AI Centre, Dept. of Computer Science, University College London, UK（伦敦大学学院人工智能中心）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI

Comments Accepted for 26th International Conference on Artificial Intelligence in Education (AIED 2025), 22 - 26 July 2025, Palermo, Italy. 17 pages, 1 figure

2501.03181 2025-04-17 cs.SD cs.AI eess.AS 62%

FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles

Tian-Hao Zhang, Jiawei Zhang, Jun Wang, Xinyuan Qian, Xu-Cheng Yin

机构 * Tencent AI Lab（腾讯AI实验室）

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.AI、eess.AS

Comments Accepted by AAAI 2025

2504.09516 2025-04-15 cs.SD cs.CV eess.AS 62%

FSSUAVL: A Discriminative Framework using Vision Models for Federated Self-Supervised Audio and Image Understanding

Yasar Abbas Ur Rehman, Kin Wai Lau, Yuyang Xie, Ma Lan, JiaJun Shen

机构 * TCL AI Lab（TCL人工智能实验室）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、eess.AS

Comments 8 pages

2312.09736 2025-04-15 cs.CL cs.SD eess.AS 62%

HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue

Sunjae Yoon, Dahyun Kim, Eunseop Yoon, Hee Suk Yoon, Junyeong Kim, Chnag D. Yoo

机构 * Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院（KAIST））； Chung-Ang University（中央大学）

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments EMNLP 2023, 14 pages, 13 figures

2504.07229 2025-04-11 cs.CL eess.AS eess.SP 62%

Visual-Aware Speech Recognition for Noisy Scenarios

Lakshmipathi Balaji, Karan Singla

机构 * IIIT-Hyderabad（印度国际信息技术学院海得拉巴分校）； Whissle Inc.（惠斯勒公司）

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CL、eess.AS

2504.04427 2025-04-08 cs.CV cs.AI cs.HC 62%

FluentLip: A Phonemes-Based Two-stage Approach for Audio-Driven Lip Synthesis with Optical Flow Consistency

Shiyan Liu, Rui Qu, Yan Jin

机构 * Huazhong University of Science and Technology（华中科技大学）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.AI

2504.02386 2025-04-04 cs.CV eess.AS 62%

VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models

Kim Sung-Bin, Jeongsoo Choi, Puyuan Peng, Joon Son Chung, Tae-Hyun Oh, David Harwath

机构 * POSTECH（浦项科技大学）； KAIST（韩国科学技术院）； The University of Texas at Austin（德克萨斯大学奥斯汀分校）

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments https://voicecraft-dub.github.io/

2504.00750 2025-04-02 cs.SD cs.LG cs.MM eess.AS 62%

$C^2$AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction

Wenxuan Wu, Xueyuan Chen, Shuai Wang, Jiadong Wang, Lingwei Meng, Xixin Wu, Helen Meng, Haizhou Li

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.MM、eess.AS

Comments Accepted by IEEE Journal of Selected Topics in Signal Processing (JSTSP)

2503.23395 2025-04-01 cs.SD cs.AI eess.AS 62%

Scaling Auditory Cognition via Test-Time Compute in Audio Language Models

Ting Dang, Yan Gao, Hong Jia

机构 * School of Computing and Information Systems, The University of Melbourne（墨尔本大学计算与信息系统学院）； University of Cambridge（剑桥大学）

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、eess.AS

1. 音频语音多模态 4597 篇

Listen, Chat, and Remix: Text-Guided Soundscape Remixing for Enhanced Auditory Experience

MOSAIC-F: A Framework for Enhancing Students' Oral Presentation Skills through Personalized Feedback

Seeing Voices: Generating A-Roll Video from Audio with Mirage

Learning Speaker-Invariant Visual Features for Lipreading

Bridging Audio and Vision: Zero-Shot Audiovisual Segmentation by Connecting Pretrained Models

A Pre-trained Framework for Multilingual Brain Decoding Using Non-invasive Recordings

Transformers in Speech Processing: A Survey

Chain-of-Thought Training for Open E2E Spoken Dialogue Systems

Spatiotemporal Emotional Synchrony in Dyadic Interactions: The Role of Speech Conditions in Facial and Vocal Affective Alignment

SparQLe: Speech Queries to Text Translation Through LLMs

Spoken question answering for visual queries

Assessment of L2 Oral Proficiency using Speech Large Language Models

DeepDialogue: A Multi-Turn Emotionally-Rich Spoken Dialogue Dataset

vec2wav 2.0: Advancing Voice Conversion via Discrete Token Vocoders

UniTTS: An end-to-end TTS system without decoupling of acoustic and semantic information

ToxicTone: A Mandarin Audio Dataset Annotated for Toxicity and Toxic Utterance Tonality

In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties

Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals

A vector quantized masked autoencoder for audiovisual speech emotion recognition

VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling

APG-MOS: Auditory Perception Guided-MOS Predictor for Synthetic Speech

Rethinking the Potential of Multimodality in Collaborative Problem Solving Diagnosis with Large Language Models

FaceSpeak: Expressive and High-Quality Speech Synthesis from Human Portraits of Different Styles

FSSUAVL: A Discriminative Framework using Vision Models for Federated Self-Supervised Audio and Image Understanding

HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue

Visual-Aware Speech Recognition for Noisy Scenarios

FluentLip: A Phonemes-Based Two-stage Approach for Audio-Driven Lip Synthesis with Optical Flow Consistency

VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models

$C^2$AV-TSE: Context and Confidence-aware Audio Visual Target Speaker Extraction

Scaling Auditory Cognition via Test-Time Compute in Audio Language Models