Evaluation of conversational naturalness is essential for developing human-like speech agents. However, existing speech naturalness predictors are often designed to assess utterances from a single speaker, failing to capture conversation-level naturalness qualities. In this paper, we present a framework for an automatic naturalness predictor for two-speaker, multi-turn conversations. We first show that existing naturalness estimators have low, or sometimes even negative, correlations with conversational naturalness, based on conversational recordings annotated with human ratings. We then propose a dual-channel naturalness estimator, in which we investigate multiple pre-trained encoders with data augmentation. Our proposed model achieves substantially higher correlation with human judgments compared to existing naturalness predictors for both in-domain and out-of-domain conditions.
The T05 System for The VoiceMOS Challenge 2024: Transfer Learning from Deep Image Classifier to Naturalness MOS Prediction of High-Quality Synthetic Speech
DiffCSS: Diverse and Expressive Conversational Speech Synthesis with Diffusion Models
Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院);The Chinese University of Hong Kong(香港中文大学);Tencent Youtu Lab(腾讯优图实验室);StepFun(阶跃星辰)
Beijing University of Posts and Telecommunications(北京邮电大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Unisound AI Technology Co., Ltd(云知声智能科技股份有限公司)
Speaker Role and Language Diarization for Analyzing Multilingual Interviews for Language Proficiency of Older Adults
Viterbi School of Engineering, University of Southern California(南加州大学维特比工程学院);Center for Economic and Social Research, University of Southern California(南加州大学经济社会研究中心);Columbia University Irving Medical Center(哥伦比亚大学欧文医学中心)
Towards Building Voice-based Conversational Recommender Systems: Datasets, Potential Solutions, and Prospects
Bytedance AI Lab(字节跳动人工智能实验室);Institute of High Performance Computing, Centre for Frontier AI Research, A*STAR(新加坡科技研究局前沿人工智能研究中心高性能计算研究所);A*STAR Centre for Frontier AI Research(新加坡科技研究局前沿人工智能研究中心);Nanyang Technological University(南洋理工大学)
Music Discovery Dialogue Generation Using Human Intent Analysis and Large Language Models
Graduate School of Culture Technology, KAIST(韩国科学技术院文化技术研究生院);Prescient Design, Genentech(基因泰克公司Prescient Design部门);Department of Industrial Design, KAIST(韩国科学技术院工业设计系)
NTPP: Generative Speech Language Modeling for Dual-Channel Spoken Dialogue via Next-Token-Pair Prediction
Tencent(腾讯);National University of Singapore(新加坡国立大学);The Chinese University of Hong Kong(香港中文大学);Nanyang Technological University(南洋理工大学);Shenzhen University(深圳大学)
SponTTS: modeling and transferring spontaneous style for TTS
Northwestern Polytechnical University(西北工业大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shanghai Mobvoi Information Technology Co., Ltd(上海出门问问信息技术有限公司)