多模态大模型 - arXivDaily 专题

1909.02764 2019-09-10 cs.CL cs.HC eess.AS 76%

Towards Multimodal Emotion Recognition in German Speech Events in Cars using Transfer Learning

Deniz Cevher, Sebastian Zepf, Roman Klinger

专题命中音频语音多模态：multimodal(title)；分类 cs.CL、eess.AS

Comments 12 pages, 2 figures, accepted at KONVENS 2019

1907.04355 2019-07-11 cs.CL cs.LG cs.SD eess.AS 76%

Transfer Learning from Audio-Visual Grounding to Speech Recognition

Wei-Ning Hsu, David Harwath, James Glass

专题命中音频语音多模态：audio-visual(title)；分类 cs.CL、eess.AS

Comments Accepted to Interspeech 2019. 4 pages, 2 figures

1812.09336 2018-12-27 cs.CV cs.CL 76%

An Empirical Analysis of Deep Audio-Visual Models for Speech Recognition

Devesh Walawalkar, Yihui He, Rohit Pillai

专题命中音频语音多模态：audio-visual(title)；分类 cs.CV、cs.CL

1812.08318 2018-12-21 cs.CL cs.SD eess.AS 76%

Generating lyrics with variational autoencoder and multi-modal artist embeddings

Olga Vechtomova, Hareesh Bahuleyan, Amirpasha Ghabussi, Vineet John

专题命中音频语音多模态：multi-modal(title)；分类 cs.CL、eess.AS

Comments 5 pages, 5 tables, 1 figure

1811.11623 2018-11-29 cs.AI cs.CV 76%

Large Scale Audio-Visual Video Analytics Platform for Forensic Investigations of Terroristic Attacks

Alexander Schindler, Martin Boyer, Andrew Lindley, David Schreiber, Thomas Philipp

专题命中音频语音多模态：audio-visual(title)；分类 cs.CV、cs.AI

Journal ref 25th International Conference on MultiMedia Modeling (MMM2019)

1808.00060 2018-09-12 cs.SD cs.CV cs.LG eess.AS 76%

DNN driven Speaker Independent Audio-Visual Mask Estimation for Speech Separation

Mandar Gogate, Ahsan Adeel, Ricard Marxer, Jon Barker, Amir Hussain

专题命中音频语音多模态：audio-visual(title)；分类 cs.CV、eess.AS

Comments Accepted for Interspeech 2018, 5 pages, 4 figures

1808.06288 2018-08-21 eess.AS cs.CL cs.SD stat.ML 76%

Multimodal speech synthesis architecture for unsupervised speaker adaptation

Hieu-Thi Luong, Junichi Yamagishi

专题命中音频语音多模态：multimodal(title)；分类 cs.CL、eess.AS

Comments Accepted for Interspeech 2018, India

1804.09713 2018-04-27 eess.AS cs.CL cs.LG 76%

End-to-End Multimodal Speech Recognition

Shruti Palaskar, Ramon Sanabria, Florian Metze

专题命中音频语音多模态：multimodal(title)；分类 cs.CL、eess.AS

Comments 5 pages, 5 figures, Accepted at IEEE International Conference on Acoustics, Speech and Signal Processing 2018 (ICASSP 2018)

1704.08619 2018-02-14 cs.CV cs.CL 76%

End-to-End Multimodal Emotion Recognition using Deep Neural Networks

Panagiotis Tzirakis, George Trigeorgis, Mihalis A. Nicolaou, Björn Schuller, Stefanos Zafeiriou

专题命中音频语音多模态：multimodal(title)；分类 cs.CV、cs.CL

1712.02896 2017-12-11 cs.CV cs.CL 76%

Audio-Visual Sentiment Analysis for Learning Emotional Arcs in Movies

Eric Chu, Deb Roy

专题命中音频语音多模态：audio-visual(title)；分类 cs.CV、cs.CL

Comments Data Mining (ICDM), 2017 IEEE 17th International Conference on

1701.07481 2017-05-26 cs.CL cs.CV 76%

Learning Word-Like Units from Joint Audio-Visual Analysis

David Harwath, James R. Glass

专题命中音频语音多模态：audio-visual(title)；分类 cs.CV、cs.CL

2603.15685 2026-07-09 cs.MM cs.AI cs.CV cs.SD 版本更新 75%

DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression

DASH：动态音频驱动的语义分块以实现高效的多模态令牌压缩

Bingzhou Li, Tao Huang

机构 * Shanghai Jiao Tong University（上海交通大学）； Tongji University（同济大学）

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI、cs.MM

AI总结 DASH通过动态音频驱动的语义分块方法，有效压缩多模态令牌，提升推理效率，优于传统固定窗口分块和注意力剪枝方法。

Comments ECCV 2026

2604.22209 2026-04-27 eess.AS cs.AI cs.CL cs.SD 75%

UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions

UniSonate：一种统一的语音、音乐和音效生成模型，通过文本指令控制

Chunyu Qiang, Xiaopeng Wang, Kang Yin, Yuzhe Liang, Yuxin Guo, Teng Ma, Ziyu Zhang, Tianrui Wang, Cheng Gong, Yushen Chen, Ruibo Fu, Chen Zhang, Longbiao Wang, Jianwu Dang

机构 * Tianjin University（天津大学）； Kling Team, Kuaishou Technology（快手科技 Kling 团队）； Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

AI总结 UniSonate通过统一的文本指令接口生成语音、音乐和音效，采用动态令牌注入机制和多阶段课程学习策略，提升跨模态生成性能，实现指令驱动的TTS和TTM的SOTA表现。

Comments Accepted to ACL 2026 main conference (oral)

2503.10522 2026-04-16 cs.MM cs.CV cs.LG cs.SD eess.AS 75%

AudioX: A Unified Framework for Anything-to-Audio Generation

AudioX：一种用于任意到音频生成的统一框架

Zeyue Tian, Zhaoyang Liu, Yizhu Jin, Ruibin Yuan, Liumeng Xue, Xu Tan, Qifeng Chen, Wei Xue, Yike Guo

机构 * Hong Kong University of Science and Technology（香港科技大学）； Independent Researcher（独立研究者）

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

AI总结本文提出AudioX框架，整合多种模态条件，通过多模态自适应融合模块提升生成质量，并构建大规模高质量数据集IF-caps，验证了其在文本到音频和文本到音乐生成中的优越性能。

Comments Accepted to ICLR 2026

2505.10101 2026-04-16 cs.SD cs.AI cs.GR cs.MM eess.AS 75%

LAV: Audio-Driven Dynamic Visual Generation with Neural Compression and StyleGAN2

LAV：基于神经压缩和StyleGAN2的音频驱动动态视觉生成

Jongmin Jung, Dasaem Jeong

机构 * Dept. of Artificial Intelligence（人工智能系）； Dept. of Art & Technology（艺术与技术系）

专题命中音频语音多模态：audio-visual(abstract,abstract_cn)；分类 cs.AI、cs.MM、eess.AS

AI总结 LAV结合EnCodec神经音频压缩与StyleGAN2生成能力，通过随机初始化线性映射将音频嵌入转换为StyleGAN2的风格潜在空间，实现语义丰富的音频-视觉转换。

Comments Paper accepted at ISEA 2025, The 30th International Symposium on Electronic/Emerging Art, Seoul, Republic of Korea, 23 - 29 May 2025

Journal ref Proceedings of the International Symposium on Electronic/Emerging Art: 2025, Seoul, Republic of Korea / no., 2025, pp.1129-1132

2601.04563 2026-01-14 cs.LG cs.AI cs.CL cs.CV 75%

A Vision for Multisensory Intelligence: Sensing, Science, and Synergy

多感官智能的愿景：感知、科学与协同

Paul Pu Liang

机构 * MIT Media Lab and MIT EECS（MIT媒体实验室和MIT电子工程与计算机科学系）

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结本文提出未来十年多感官人工智能的研究愿景，强调通过感知、科学和协同三大主题推动多感官技术发展，提升人与AI的交互体验。

2510.24103 2025-10-29 cs.SD cs.AI cs.MM eess.AS 75%

Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation

Kang Zhang, Trung X. Pham, Suyeon Lee, Axi Niu, Arda Senocak, Joon Son Chung

机构 * KAIST, South Korea（韩国釜山国立大学）； NWPU, China（中国西北工业大学）； UNIST, South Korea（韩国全南国立大学）

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments accepted by NeurIPS 2025

2510.11738 2025-10-15 cs.SD cs.AI cs.CV cs.MM 75%

SeeingSounds: Learning Audio-to-Visual Alignment via Text

Simone Carnemolla, Matteo Pennisi, Chiara Russo, Simone Palazzo, Daniela Giordano, Concetto Spampinato

机构 * University of Catania（卡塔尼亚大学）

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments accepted to ACM Multimedia Asia 2025

2509.04606 2025-09-08 cs.CL cs.AI cs.CV 75%

Sample-efficient Integration of New Modalities into Large Language Models

Osman Batur İnce, André F. T. Martins, Oisin Mac Aodha, Edoardo M. Ponti

机构 * University of Edinburgh（爱丁堡大学）； Instituto de Telecomunicações（电信研究所）； Instituto Superior Técnico, Universidade de Lisboa（里斯本大学技术学院）； Unbabel

专题命中音频语音多模态：multimodal(abstract)；multimodal foundation model(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Pre-print

2411.15447 2025-08-13 cs.MM cs.CV cs.SD eess.AS 75%

Gotta Hear Them All: Towards Sound Source Aware Audio Generation

Wei Guo, Heng Wang, Jianbo Ma, Weidong Cai

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 17 pages, 12 figures, source code available at https://github.com/wguo86/SSV2A

2503.11026 2025-07-31 eess.AS cs.CV cs.LG cs.MM 75%

MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual Translation

Sungwoo Cho, Jeongsoo Choi, Sungnyun Kim, Se-Young Yun

机构 * KAIST AI（韩国科学技术院人工智能研究所）； KAIST EE（韩国科学技术院电子工程系）

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to ICCV 2025

2506.04214 2025-06-05 cs.CV cs.LG cs.MM cs.SD eess.AS 75%

Sounding that Object: Interactive Object-Aware Image to Audio Generation

Tingle Li, Baihe Huang, Xiaobin Zhuang, Dongya Jia, Jiawei Chen, Yuping Wang, Zhuo Chen, Gopala Anumanchipalli, Yuxuan Wang

专题命中音频语音多模态：multi-modal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments ICML 2025

2506.01319 2025-06-03 cs.SD cs.CV cs.MM eess.AS 75%

Learning Sparsity for Effective and Efficient Music Performance Question Answering

Xingjian Diao, Tianzhen Yang, Chunhui Zhang, Weiyi Wu, Ming Cheng, Jiang Gui

机构 * Dartmouth College（达特茅斯学院）； Yale University（耶鲁大学）

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to the main conference of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

2505.14336 2025-05-22 eess.AS cs.CV cs.MM cs.SD 75%

Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach

Umberto Cappellazzo, Minsu Kim, Stavros Petridis, Daniele Falavigna, Alessio Brutti

机构 * Imperial College London（伦敦帝国学院）； Meta AI ； Fondazione Bruno Kessler（布鲁诺·克塞尔基金会）

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Interspeech 2025

2505.11020 2025-05-19 cs.CV cs.MM cs.SD eess.AS 75%

Classifying Shelf Life Quality of Pineapples by Combining Audio and Visual Features

Yi-Lu Jiang, Wen-Chang Chang, Ching-Lin Wang, Kung-Liang Hsu, Chih-Yi Chiu

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

2504.01094 2025-04-03 cs.SD cs.AI cs.CL cs.CR eess.AS 75%

Multilingual and Multi-Accent Jailbreaking of Audio LLMs

Jaechul Roh, Virat Shejwalkar, Amir Houmansadr

专题命中音频语音多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 21 pages, 6 figures, 15 tables

2502.06710 2025-02-11 cs.CV cs.MM cs.SD eess.AS 75%

Learning Musical Representations for Music Performance Question Answering

Xingjian Diao, Chunhui Zhang, Tingxuan Wu, Ming Cheng, Zhongyu Ouyang, Weiyi Wu, Jiang Gui

专题命中音频语音多模态：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted at EMNLP 2024

2411.00774 2024-12-10 cs.SD cs.AI cs.CL eess.AS 75%

Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM

Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, Long Ma

专题命中音频语音多模态：multimodal(abstract)；multi-modal(abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Project Page: https://freeze-omni.github.io/

2410.21269 2024-10-29 cs.SD cs.CV cs.MM eess.AS 75%

OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup

Xize Cheng, Siqi Zheng, Zehan Wang, Minghui Fang, Ziang Zhang, Rongjie Huang, Ziyang Ma, Shengpeng Ji, Jialong Zuo, Tao Jin, Zhou Zhao

专题命中音频语音多模态：multi-modal(abstract)；omni-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Working in progress

2409.13689 2024-09-23 cs.CV cs.MM cs.SD eess.AS 75%

Temporally Aligned Audio for Video with Autoregression

Ilpo Viertola, Vladimir Iashin, Esa Rahtu

专题命中音频语音多模态：cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Submitted to ICASSP 2025. Project page https://v-aura.notion.site

1. 音频语音多模态 4587 篇

Towards Multimodal Emotion Recognition in German Speech Events in Cars using Transfer Learning

Transfer Learning from Audio-Visual Grounding to Speech Recognition

An Empirical Analysis of Deep Audio-Visual Models for Speech Recognition

Generating lyrics with variational autoencoder and multi-modal artist embeddings

Large Scale Audio-Visual Video Analytics Platform for Forensic Investigations of Terroristic Attacks

DNN driven Speaker Independent Audio-Visual Mask Estimation for Speech Separation

Multimodal speech synthesis architecture for unsupervised speaker adaptation

End-to-End Multimodal Speech Recognition

End-to-End Multimodal Emotion Recognition using Deep Neural Networks

Audio-Visual Sentiment Analysis for Learning Emotional Arcs in Movies

Learning Word-Like Units from Joint Audio-Visual Analysis

DASH: Dynamic Audio-Driven Semantic Chunking for Efficient Omnimodal Token Compression

UniSonate: A Unified Model for Speech, Music, and Sound Effect Generation with Text Instructions

AudioX: A Unified Framework for Anything-to-Audio Generation

LAV: Audio-Driven Dynamic Visual Generation with Neural Compression and StyleGAN2

A Vision for Multisensory Intelligence: Sensing, Science, and Synergy

Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generation

SeeingSounds: Learning Audio-to-Visual Alignment via Text

Sample-efficient Integration of New Modalities into Large Language Models

Gotta Hear Them All: Towards Sound Source Aware Audio Generation

MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual Translation

Sounding that Object: Interactive Object-Aware Image to Audio Generation

Learning Sparsity for Effective and Efficient Music Performance Question Answering

Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approach

Classifying Shelf Life Quality of Pineapples by Combining Audio and Visual Features

Multilingual and Multi-Accent Jailbreaking of Audio LLMs

Learning Musical Representations for Music Performance Question Answering

Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM

OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup

Temporally Aligned Audio for Video with Autoregression