Paralinguistics-Aware Speech-Empowered Large Language Models for Natural Conversation
Seoul National University（首尔大学）；NAVER Cloud（NAVER云）；Artificial Intelligence Institute, Seoul National University（首尔大学人工智能研究院）；NAVER AI Lab（NAVER人工智能实验室）
Heeseung Kim, Soonshin Seo, Kyeongseok Jeong, Ohsung Kwon, Soyoon Kim, Jungwhan Kim, Jaehong Lee, Eunwoo Song, Myungwoo Oh, Jung-Woo Ha, Sungroh Yoon, Kang Min Yoo
Recent work shows promising results in expanding the capabilities of large language models (LLM) to directly understand and synthesize speech. However, an LLM-based strategy for modeling spoken dialogs remains elusive, calling for further investigation. This paper introduces an extensive speech-text LLM framework, the Unified Spoken Dialog Model (USDM), designed to generate coherent spoken responses with naturally occurring prosodic features relevant to the given input speech without relying on explicit automatic speech recognition (ASR) or text-to-speech (TTS) systems. We have verified the inclusion of prosody in speech tokens that predominantly contain semantic information and have used this foundation to construct a prosody-infused speech-text model. Additionally, we propose a generalized speech-text pretraining scheme that enhances the capture of cross-modal semantics. To construct USDM, we fine-tune our speech-text model on spoken dialog data using a multi-step spoken dialog template that stimulates the chain-of-reasoning capabilities exhibited by the underlying LLM. Automatic and human evaluations on the DailyTalk dataset demonstrate that our approach effectively generates natural-sounding spoken responses, surpassing previous and cascaded baselines. Our code and checkpoints are available at https://github.com/naver-ai/usdm.
University of Oxford（牛津大学）；The Alan Turing Institute（艾伦·图灵研究所）；University of Sheffield（谢菲尔德大学）；Queen Mary University of London（伦敦玛丽女王大学）
Bipolar disorder (BD) and borderline personality disorder (BPD) are two chronic mental health conditions that clinicians find challenging to distinguish based on clinical interviews, due to their overlapping symptoms. In this work, we investigate the automatic detection of these two conditions by modelling both verbal and non-verbal cues in a set of interviews. We propose a new approach of modelling short-term features with visibility-signature transform, and compare it with widely used high-level statistical functions. We demonstrate the superior performance of our proposed signature-based mo…
SynParaSpeech: Automated Synthesis of Paralinguistic Datasets for Speech Generation and Understanding
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, China（北京邮电大学人工智能学院）；Hello Group Inc., China（Hello集团有限公司）；Institute of Automation, Chinese Academy of Sciences, China（中国科学院自动化研究所）
Speech-based Psychological Crisis Assessment using LLMs
Tsinghua University（清华大学）；Peking University Huilongguan Clinical Medical School（北京大学回龙guan临床医学院）；WHO Collaborating Centre for Research and Training in Suicide Prevention（世界卫生组织自杀预防研究与培训协作中心）
StyleBench: Evaluating Speech Language Models on Conversational Speaking Style Control
NLP Lab, School of Computer Science and Engineering, Northeastern University, Shenyang, China（计算机科学与工程学院自然语言处理实验室，东北大学，沈阳，中国）；NiuTrans Research, Shenyang, China（尼乌透研究，沈阳，中国）
Meta AI Research（元人工智能研究院）；EHESS（法国高等社会科学研究院）；Inria（法国国家信息与自动化研究所）
We introduce dGSLM, the first "textless" model able to generate audio samples of naturalistic spoken dialogues. It uses recent work on unsupervised spoken unit discovery coupled with a dual-tower transformer architecture with cross-attention trained on 2000 hours of two-channel raw conversational audio (Fisher dataset) without any text or labels. We show that our model is able to generate speech, laughter and other paralinguistic signals in the two channels simultaneously and reproduces more naturalistic and fluid turn-taking compared to a text-based cascaded model.
Sigma Technologies S.L.U.（西格玛技术有限公司）；Universidad Politécnica de Madrid（马德里理工大学）
The fast increase of web services and mobile apps, which collect personal data from users, increases the risk that their privacy may be severely compromised. In particular, the increasing variety of spoken language interfaces and voice assistants empowered by the vertiginous breakthroughs in Deep Learning are prompting important concerns in the European Union to preserve speech data privacy. For instance, an attacker can record speech from users and impersonate them to get access to systems requiring voice identification. Hacking speaker profiles from users is also possible by means of existin…
