多模态大模型 - arXivDaily 专题

2412.15838 2024-12-31 cs.AI cs.CL 62%

Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback

Jiaming Ji, Jiayi Zhou, Hantao Lou, Boyuan Chen, Donghai Hong, Xuyao Wang, Wenqi Chen, Kaile Wang, Rui Pan, Jiahao Li, Mohan Wang, Josef Dai, Tianyi Qiu, Hua Xu, Dong Li, Weipeng Chen, Jun Song, Bo Zheng, Yaodong Yang

专题命中音频语音多模态：any-to-any(abstract)；分类 cs.CL、cs.AI

2412.18733 2024-12-30 cs.CL cs.SD eess.AS 62%

Intra- and Inter-modal Context Interaction Modeling for Conversational Speech Synthesis

Zhenqi Jia, Rui Liu

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted by ICASSP 2025

2412.10117 2024-12-30 cs.SD cs.AI cs.LG eess.AS 62%

CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models

Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang, Zhijie Yan, Jingren Zhou

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.AI、eess.AS

Comments Tech report, work in progress

2412.18061 2024-12-25 cs.SD cs.CL cs.HC eess.AS 62%

Lla-VAP: LSTM Ensemble of Llama and VAP for Turn-Taking Prediction

Hyunbae Jeon, Frederic Guintu, Rayvant Sahni

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、eess.AS

2407.21315 2024-12-24 cs.CL cs.AI 62%

Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances

Zehui Wu, Ziwei Gong, Lin Ai, Pengyuan Shi, Kaan Donbekci, Julia Hirschberg

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI

2411.13674 2024-12-10 cs.CV cs.LG cs.NE cs.SD eess.AS 62%

FabuLight-ASD: Unveiling Speech Activity via Body Language

Hugo Carneiro, Stefan Wermter

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、eess.AS

Comments 23 pages, 8 figures, 3 tables, accepted for publication in Neural Computing and Applications

2412.05797 2024-12-10 cs.CL cs.AI 62%

Speech Is Not Enough: Interpreting Nonverbal Indicators of Common Knowledge and Engagement

Derek Palmer, Yifan Zhu, Kenneth Lai, Hannah VanderHoeven, Mariah Bradford, Ibrahim Khebour, Carlos Mabrey, Jack Fitzgerald, Nikhil Krishnaswamy, Martha Palmer, James Pustejovsky

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 3 pages, 2 figures, appearing at AAAI 2025 Demos Track

2412.04266 2024-12-06 cs.CL cs.SD eess.AS 62%

Representation Purification for End-to-End Speech Translation

Chengwei Zhang, Yue Zhou, Rui Zhao, Yidong Chen, Xiaodong Shi

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CL、eess.AS

Comments Accepted by COLING 2025

2412.03075 2024-12-05 cs.CL cs.SD eess.AS 62%

ASR-EC Benchmark: Evaluating Large Language Models on Chinese ASR Error Correction

Victor Junqiu Wei, Weicheng Wang, Di Jiang, Yuanfeng Song, Lu Wang

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CL、eess.AS

2411.19842 2024-12-02 eess.AS cs.AI cs.LG cs.SD eess.SP 62%

Scaling Transformers for Low-Bitrate High-Quality Speech Coding

Julian D Parker, Anton Smirnov, Jordi Pons, CJ Carr, Zack Zukowski, Zach Evans, Xubo Liu

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、eess.AS

2402.05706 2024-12-02 cs.CL cs.SD eess.AS 62%

Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation

Heeseung Kim, Soonshin Seo, Kyeongseok Jeong, Ohsung Kwon, Soyoon Kim, Jungwhan Kim, Jaehong Lee, Eunwoo Song, Myungwoo Oh, Jung-Woo Ha, Sungroh Yoon, Kang Min Yoo

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CL、eess.AS

Comments NeurIPS 2024, Project Page: https://unifiedsdm.github.io/

2410.09804 2024-11-28 cs.CR cs.AI cs.CL cs.LG cs.NE 62%

BlackDAN: A Black-Box Multi-Objective Approach for Effective and Contextual Jailbreaking of Large Language Models

Xinyuan Wang, Victor Shea-Jay Huang, Renmiao Chen, Hao Wang, Chengwei Pan, Lei Sha, Minlie Huang

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI

2406.10325 2024-11-19 cs.CL cs.LG eess.AS 62%

Enhancing Multilingual Voice Toxicity Detection with Speech-Text Alignment

Joseph Liu, Mahesh Kumar Nandwana, Janne Pylkkönen, Hannes Heikinheimo, Morgan McGuire

专题命中音频语音多模态：cross-modal(abstract)；分类 cs.CL、eess.AS

Comments Accepted to INTERSPEECH 2024

2410.05586 2024-11-12 cs.CV cs.AI 62%

TeaserGen: Generating Teasers for Long Documentaries

Weihan Xu, Paul Pu Liang, Haven Kim, Julian McAuley, Taylor Berg-Kirkpatrick, Hao-Wen Dong

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CV、cs.AI

2406.05839 2024-11-12 eess.AS cs.AI 62%

MaLa-ASR: Multimedia-Assisted LLM-Based ASR

Guanrou Yang, Ziyang Ma, Fan Yu, Zhifu Gao, Shiliang Zhang, Xie Chen

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.AI、eess.AS

2410.22271 2024-10-30 eess.AS cs.AI eess.IV eess.SP 62%

Leveraging Reverberation and Visual Depth Cues for Sound Event Localization and Detection with Distance Estimation

Davide Berghi, Philip J. B. Jackson

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.AI、eess.AS

2402.05755 2024-10-22 cs.CL cs.SD eess.AS 62%

Spirit LM: Interleaved Spoken and Written Language Model

Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Christophe Ropers, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Mary Williamson, Gabriel Synnaeve, Juan Pino, Benoit Sagot, Emmanuel Dupoux

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

2305.01661 2024-10-21 cs.SD cs.AI cs.LG eess.AS 62%

Integrating spoken instructions into flight trajectory prediction to optimize automation in air traffic control

Dongyue Guo, Zheng Zhang, Bo Yang, Jianwei Zhang, Hongyu Yang, Yi Lin

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.AI、eess.AS

Comments This paper has been accepted in principle by Nature Communications

2410.03798 2024-10-15 cs.CL cs.SD eess.AS 62%

Self-Powered LLM Modality Expansion for Large Speech-Text Models

Tengfei Yu, Xuebo Liu, Zhiyi Hou, Liang Ding, Dacheng Tao, Min Zhang

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted to EMNLP 2024

2308.00122 2024-10-14 cs.CV cs.SD eess.AS 62%

High-Quality Visually-Guided Sound Separation from Diverse Categories

Chao Huang, Susan Liang, Yapeng Tian, Anurag Kumar, Chenliang Xu

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.CV、eess.AS

Comments ACCV 2024 Oral

2207.02160 2024-10-03 cs.CL cs.AI cs.IR 62%

A Comprehensive Review of Visual-Textual Sentiment Analysis from Social Media Networks

Israa Khalaf Salman Al-Tameemi, Mohammad-Reza Feizi-Derakhshi, Saeed Pashazadeh, Mohammad Asadpour

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、cs.AI

2404.15637 2024-09-26 cs.SD cs.MM eess.AS 62%

HybridVC: Efficient Voice Style Conversion with Text and Audio Prompts

Xinlei Niu, Jing Zhang, Charles Patrick Martin

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.MM、eess.AS

Comments Proceedings of Interspeech

Journal ref Proc. Interspeech 2024, 4368-4372

2409.16203 2024-09-25 cs.SD cs.AI eess.AS 62%

Facial Expression-Enhanced TTS: Combining Face Representation and Emotion Intensity for Adaptive Speech

Yunji Chu, Yunseob Shim, Unsang Park

专题命中音频语音多模态：audio-visual(abstract)；分类 cs.AI、eess.AS

Comments 13 pages, 3 figures, accepted to ECCV Workshop ABAW(Affective Behavior Analysis in-the-wild)7 (to be appear)

2409.16005 2024-09-25 cs.CL cs.SD eess.AS 62%

Bridging Speech and Text: Enhancing ASR with Pinyin-to-Character Pre-training in LLMs

Yang Yuhang, Peng Yizhou, Eng Siong Chng, Xionghu Zhong

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted by ISCSLP2024-Special session-Speech Processing in LLM Era

2409.15353 2024-09-25 eess.AS cs.CL cs.LG cs.SD 62%

Contextualization of ASR with LLM using phonetic retrieval-based augmentation

Zhihong Lei, Xingyu Na, Mingbin Xu, Ernest Pusateri, Christophe Van Gysel, Yuanyuan Zhang, Shiyi Han, Zhen Huang

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

2402.13276 2024-09-25 eess.AS cs.AI cs.SD 62%

When LLMs Meets Acoustic Landmarks: An Efficient Approach to Integrate Speech into Large Language Models for Depression Detection

Xiangyu Zhang, Hexin Liu, Kaishuai Xu, Qiquan Zhang, Daijiao Liu, Beena Ahmed, Julien Epps

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、eess.AS

2311.14517 2024-09-25 cs.SD cs.CL cs.LG eess.AS 62%

tinyCLAP: Distilling Constrastive Language-Audio Pretrained Models

Francesco Paissan, Elisabetta Farella

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Proceedings of Interspeech. Please use the citation available at https://www.isca-archive.org/interspeech_2024/paissan24_interspeech.html

2405.02151 2024-09-24 cs.SD cs.AI eess.AS 62%

GMP-TL: Gender-augmented Multi-scale Pseudo-label Enhanced Transfer Learning for Speech Emotion Recognition

Yu Pan, Yuguang Yang, Heng Lu, Lei Ma, Jianjun Zhao

专题命中音频语音多模态：multimodal(abstract)；分类 cs.AI、eess.AS

Comments Accepted to SLT2024

2409.13557 2024-09-23 cs.CV cs.AI 62%

Trustworthy Hate Speech Detection Through Visual Augmentation

Ziyuan Yang, Ming Yan, Yingyu Chen, Hui Wang, Zexin Lu, Yi Zhang

专题命中音频语音多模态：multi-modal(abstract)；分类 cs.CV、cs.AI

2306.01442 2024-09-19 cs.SD cs.CL cs.LG eess.AS 62%

Towards Robust FastSpeech 2 by Modelling Residual Multimodality

Fabian Kögel, Bac Nguyen, Fabien Cardinaux

专题命中音频语音多模态：multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted at INTERSPEECH 2023

1. 音频语音多模态 4597 篇

Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback

Intra- and Inter-modal Context Interaction Modeling for Conversational Speech Synthesis

CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models

Lla-VAP: LSTM Ensemble of Llama and VAP for Turn-Taking Prediction

Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances

FabuLight-ASD: Unveiling Speech Activity via Body Language

Speech Is Not Enough: Interpreting Nonverbal Indicators of Common Knowledge and Engagement

Representation Purification for End-to-End Speech Translation

ASR-EC Benchmark: Evaluating Large Language Models on Chinese ASR Error Correction

Scaling Transformers for Low-Bitrate High-Quality Speech Coding

Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation

BlackDAN: A Black-Box Multi-Objective Approach for Effective and Contextual Jailbreaking of Large Language Models

Enhancing Multilingual Voice Toxicity Detection with Speech-Text Alignment

TeaserGen: Generating Teasers for Long Documentaries

MaLa-ASR: Multimedia-Assisted LLM-Based ASR

Leveraging Reverberation and Visual Depth Cues for Sound Event Localization and Detection with Distance Estimation

Spirit LM: Interleaved Spoken and Written Language Model

Integrating spoken instructions into flight trajectory prediction to optimize automation in air traffic control

Self-Powered LLM Modality Expansion for Large Speech-Text Models

High-Quality Visually-Guided Sound Separation from Diverse Categories

A Comprehensive Review of Visual-Textual Sentiment Analysis from Social Media Networks

HybridVC: Efficient Voice Style Conversion with Text and Audio Prompts

Facial Expression-Enhanced TTS: Combining Face Representation and Emotion Intensity for Adaptive Speech

Bridging Speech and Text: Enhancing ASR with Pinyin-to-Character Pre-training in LLMs

Contextualization of ASR with LLM using phonetic retrieval-based augmentation

When LLMs Meets Acoustic Landmarks: An Efficient Approach to Integrate Speech into Large Language Models for Depression Detection

tinyCLAP: Distilling Constrastive Language-Audio Pretrained Models

GMP-TL: Gender-augmented Multi-scale Pseudo-label Enhanced Transfer Learning for Speech Emotion Recognition

Trustworthy Hate Speech Detection Through Visual Augmentation

Towards Robust FastSpeech 2 by Modelling Residual Multimodality