多模态大模型 - arXivDaily 专题

2412.03075 2024-12-05 cs.CL cs.SD eess.AS 62%

ASR-EC Benchmark: Evaluating Large Language Models on Chinese ASR Error Correction

Victor Junqiu Wei, Weicheng Wang, Di Jiang, Yuanfeng Song, Lu Wang

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、eess.AS

2411.19842 2024-12-02 eess.AS cs.AI cs.LG cs.SD eess.SP 62%

Scaling Transformers for Low-Bitrate High-Quality Speech Coding

Julian D Parker, Anton Smirnov, Jordi Pons, CJ Carr, Zack Zukowski, Zach Evans, Xubo Liu

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、eess.AS

2402.05706 2024-12-02 cs.CL cs.SD eess.AS 62%

Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation

Heeseung Kim, Soonshin Seo, Kyeongseok Jeong, Ohsung Kwon, Soyoon Kim, Jungwhan Kim, Jaehong Lee, Eunwoo Song, Myungwoo Oh, Jung-Woo Ha, Sungroh Yoon, Kang Min Yoo

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CL、eess.AS

Comments NeurIPS 2024, Project Page: https://unifiedsdm.github.io/

2410.09804 2024-11-28 cs.CR cs.AI cs.CL cs.LG cs.NE 62%

BlackDAN: A Black-Box Multi-Objective Approach for Effective and Contextual Jailbreaking of Large Language Models

Xinyuan Wang, Victor Shea-Jay Huang, Renmiao Chen, Hao Wang, Chengwei Pan, Lei Sha, Minlie Huang

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI

2406.10325 2024-11-19 cs.CL cs.LG eess.AS 62%

Enhancing Multilingual Voice Toxicity Detection with Speech-Text Alignment

Joseph Liu, Mahesh Kumar Nandwana, Janne Pylkkönen, Hannes Heikinheimo, Morgan McGuire

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CL、eess.AS

Comments Accepted to INTERSPEECH 2024

2410.05586 2024-11-12 cs.CV cs.AI 62%

TeaserGen: Generating Teasers for Long Documentaries

Weihan Xu, Paul Pu Liang, Haven Kim, Julian McAuley, Taylor Berg-Kirkpatrick, Hao-Wen Dong

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.AI

2406.05839 2024-11-12 eess.AS cs.AI 62%

MaLa-ASR: Multimedia-Assisted LLM-Based ASR

Guanrou Yang, Ziyang Ma, Fan Yu, Zhifu Gao, Shiliang Zhang, Xie Chen

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.AI、eess.AS

2410.22271 2024-10-30 eess.AS cs.AI eess.IV eess.SP 62%

Leveraging Reverberation and Visual Depth Cues for Sound Event Localization and Detection with Distance Estimation

Davide Berghi, Philip J. B. Jackson

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.AI、eess.AS

2402.05755 2024-10-22 cs.CL cs.SD eess.AS 62%

Spirit LM: Interleaved Spoken and Written Language Model

Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoit Sagot, Emmanuel Dupoux

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

2305.01661 2024-10-21 cs.SD cs.AI cs.LG eess.AS 62%

Integrating spoken instructions into flight trajectory prediction to optimize automation in air traffic control

Dongyue Guo, Zheng Zhang, Bo Yang, Jianwei Zhang, Hongyu Yang, Yi Lin

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.AI、eess.AS

Comments This paper has been accepted in principle by Nature Communications

2410.03798 2024-10-15 cs.CL cs.SD eess.AS 62%

Self-Powered LLM Modality Expansion for Large Speech-Text Models

Tengfei Yu, Xuebo Liu, Zhiyi Hou, Liang Ding, Dacheng Tao, Min Zhang

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted to EMNLP 2024

2308.00122 2024-10-14 cs.CV cs.SD eess.AS 62%

High-Quality Visually-Guided Sound Separation from Diverse Categories

Chao Huang, Susan Liang, Yapeng Tian, Anurag Kumar, Chenliang Xu

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments ACCV 2024 Oral

2207.02160 2024-10-03 cs.CL cs.AI cs.IR 62%

A Comprehensive Review of Visual-Textual Sentiment Analysis from Social Media Networks

Israa Khalaf Salman Al-Tameemi, Mohammad-Reza Feizi-Derakhshi, Saeed Pashazadeh, Mohammad Asadpour

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI

2404.15637 2024-09-26 cs.SD cs.MM eess.AS 62%

HybridVC: Efficient Voice Style Conversion with Text and Audio Prompts

Xinlei Niu, Jing Zhang, Charles Patrick Martin

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.MM、eess.AS

Comments Proceedings of Interspeech

Journal ref Proc. Interspeech 2024, 4368-4372

2409.16203 2024-09-25 cs.SD cs.AI eess.AS 62%

Facial Expression-Enhanced TTS: Combining Face Representation and Emotion Intensity for Adaptive Speech

Yunji Chu, Yunseob Shim, Unsang Park

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.AI、eess.AS

Comments 13 pages, 3 figures, accepted to ECCV Workshop ABAW(Affective Behavior Analysis in-the-wild)7 (to be appear)

2409.16005 2024-09-25 cs.CL cs.SD eess.AS 62%

Bridging Speech and Text: Enhancing ASR with Pinyin-to-Character Pre-training in LLMs

Yang Yuhang, Peng Yizhou, Eng Siong Chng, Xionghu Zhong

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted by ISCSLP2024-Special session-Speech Processing in LLM Era

2409.15353 2024-09-25 eess.AS cs.CL cs.LG cs.SD 62%

Contextualization of ASR with LLM using phonetic retrieval-based augmentation

Zhihong Lei, Xingyu Na, Mingbin Xu, Ernest Pusateri, Christophe Van Gysel, Yuanyuan Zhang, Shiyi Han, Zhen Huang

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

2402.13276 2024-09-25 eess.AS cs.AI cs.SD 62%

When LLMs Meets Acoustic Landmarks: An Efficient Approach to Integrate Speech into Large Language Models for Depression Detection

Xiangyu Zhang, Hexin Liu, Kaishuai Xu, Qiquan Zhang, Daijiao Liu, Beena Ahmed, Julien Epps

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、eess.AS

2311.14517 2024-09-25 cs.SD cs.CL cs.LG eess.AS 62%

tinyCLAP: Distilling Constrastive Language-Audio Pretrained Models

Francesco Paissan, Elisabetta Farella

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Proceedings of Interspeech. Please use the citation available at https://www.isca-archive.org/interspeech_2024/paissan24_interspeech.html

2405.02151 2024-09-24 cs.SD cs.AI eess.AS 62%

GMP-TL: Gender-augmented Multi-scale Pseudo-label Enhanced Transfer Learning for Speech Emotion Recognition

Yu Pan, Yuguang Yang, Heng Lu, Lei Ma, Jianjun Zhao

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、eess.AS

Comments Accepted to SLT2024

2409.13557 2024-09-23 cs.CV cs.AI 62%

Trustworthy Hate Speech Detection Through Visual Augmentation

Ziyuan Yang, Ming Yan, Yingyu Chen, Hui Wang, Zexin Lu, Yi Zhang

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CV、cs.AI

2306.01442 2024-09-19 cs.SD cs.CL cs.LG eess.AS 62%

Towards Robust FastSpeech 2 by Modelling Residual Multimodality

Fabian Kögel, Bac Nguyen, Fabien Cardinaux

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted at INTERSPEECH 2023

2409.11449 2024-09-19 cs.LG cs.AI cs.IR cs.SD eess.AS 62%

Evaluation of pretrained language models on music understanding

Yannis Vasilakis, Rachel Bittner, Johan Pauwels

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、eess.AS

2409.09823 2024-09-17 cs.SD cs.MM eess.AS 62%

Efficient Video to Audio Mapper with Visual Scene Detection

Mingjing Yi, Ming Li

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.MM、eess.AS

2409.09289 2024-09-17 cs.SD cs.MM eess.AS 62%

DSCLAP: Domain-Specific Contrastive Language-Audio Pre-Training

Shengqiang Liu, Da Liu, Anna Wang, Zhiyu Zhang, Jie Gao, Yali Li

专题命中音频语音多模态：multimodal(abstract)；分类 cs.MM、eess.AS

2409.09213 2024-09-17 eess.AS cs.CL cs.SD 62%

ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds

Sreyan Ghosh, Sonal Kumar, Chandra Kiran Reddy Evuru, Oriol Nieto, Ramani Duraiswami, Dinesh Manocha

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、eess.AS

Comments Code and Checkpoints: https://github.com/Sreyan88/ReCLAP

2409.08795 2024-09-17 eess.AS cs.MM 62%

LLaQo: Towards a Query-Based Coach in Expressive Music Performance Assessment

Huan Zhang, Vincent Cheung, Hayato Nishioka, Simon Dixon, Shinichi Furuya

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.MM、eess.AS

2409.08907 2024-09-16 cs.AI cs.CL cs.CY 62%

Affective Computing Has Changed: The Foundation Model Disruption

Björn Schuller, Adria Mallol-Ragolta, Alejandro Peña Almansa, Iosif Tsangko, Mostafa M. Amin, Anastasia Semertzidou, Lukas Christ, Shahin Amiriparian

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI

2409.06348 2024-09-11 cs.SD cs.AI cs.CR eess.AS 62%

VoiceWukong: Benchmarking Deepfake Voice Detection

Ziwei Yan, Yanjie Zhao, Haoyu Wang

专题命中音频语音多模态：MLLM(abstract)；分类 cs.AI、eess.AS

2409.03715 2024-09-06 cs.SD cs.AI eess.AS 62%

Applications and Advances of Artificial Intelligence in Music Generation:A Review

Yanxu Chen, Linshu Huang, Tian Gou

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、eess.AS

1. 音频语音多模态 4597 篇

ASR-EC Benchmark: Evaluating Large Language Models on Chinese ASR Error Correction

Scaling Transformers for Low-Bitrate High-Quality Speech Coding

Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation

BlackDAN: A Black-Box Multi-Objective Approach for Effective and Contextual Jailbreaking of Large Language Models

Enhancing Multilingual Voice Toxicity Detection with Speech-Text Alignment

TeaserGen: Generating Teasers for Long Documentaries

MaLa-ASR: Multimedia-Assisted LLM-Based ASR

Leveraging Reverberation and Visual Depth Cues for Sound Event Localization and Detection with Distance Estimation

Spirit LM: Interleaved Spoken and Written Language Model

Integrating spoken instructions into flight trajectory prediction to optimize automation in air traffic control

Self-Powered LLM Modality Expansion for Large Speech-Text Models

High-Quality Visually-Guided Sound Separation from Diverse Categories

A Comprehensive Review of Visual-Textual Sentiment Analysis from Social Media Networks

HybridVC: Efficient Voice Style Conversion with Text and Audio Prompts

Facial Expression-Enhanced TTS: Combining Face Representation and Emotion Intensity for Adaptive Speech

Bridging Speech and Text: Enhancing ASR with Pinyin-to-Character Pre-training in LLMs

Contextualization of ASR with LLM using phonetic retrieval-based augmentation

When LLMs Meets Acoustic Landmarks: An Efficient Approach to Integrate Speech into Large Language Models for Depression Detection

tinyCLAP: Distilling Constrastive Language-Audio Pretrained Models

GMP-TL: Gender-augmented Multi-scale Pseudo-label Enhanced Transfer Learning for Speech Emotion Recognition

Trustworthy Hate Speech Detection Through Visual Augmentation

Towards Robust FastSpeech 2 by Modelling Residual Multimodality

Evaluation of pretrained language models on music understanding

Efficient Video to Audio Mapper with Visual Scene Detection

DSCLAP: Domain-Specific Contrastive Language-Audio Pre-Training

ReCLAP: Improving Zero Shot Audio Classification by Describing Sounds

LLaQo: Towards a Query-Based Coach in Expressive Music Performance Assessment

Affective Computing Has Changed: The Foundation Model Disruption

VoiceWukong: Benchmarking Deepfake Voice Detection

Applications and Advances of Artificial Intelligence in Music Generation:A Review