arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.13273cs.SDcs.MMeess.AS
收藏

基于双流联合训练的非特定人发音转换算法

Two-Stream Joint-Training for Speaker Independent Acoustic-to-Articulatory Inversion

Tianjin University(天津大学);Tianjin International Engineering Institute, Tianjin University(天津大学天津国际工程学院);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Institute of Linguistics, Chinese Academy of Social Sciences(中国社会科学院语言研究所)

Jianrong Wang, Jinyu Liu, Li Liu, Xuewei Li, Mei Yu, Jie Gao, Qiang Fang

摘要:

声学发音反演(AAI)的目的是从语音音频中估计发音器官的参数。AAI面临两个共同的挑战,即有限的数据和说话人无关场景下的性能不理想。目前的研究大多集中在直接从语音中提取特征,而忽略了音素信息的重要性,这可能会限制AAI的性能。为此,我们提出了一种名为SPN的新型网络,它使用两个不同的流来执行AAI任务。首先,为了提高非特定人实验的性能,我们提出了一种新的音素流网络来估计发音参数作为音素特征。据我们所知,这是第一个工作,提取非特定人特征的音素,以提高AAI的性能。其次,为了更好地表示语音信息,我们训练了一个语音流网络,将局部特征和全局特征结合起来。在非特定人实验中,与最新的SOTA方法相比,该方法的均方根误差降低了0.18mm,皮尔逊相关系数提高了6.0%。该代码已在https://github.com/liujinyu123/AAINetwork-SPN上发布。

英文摘要:

Acoustic-to-articulatory inversion (AAI) aims to estimate the parameters of articulators from speech audio. There are two common challenges in AAI, which are the limited data and the unsatisfactory performance in speaker independent scenario. Most current works focus on extracting features directly from speech and ignoring the importance of phoneme information which may limit the performance of AAI. To this end, we propose a novel network called SPN that uses two different streams to carry out the AAI task. Firstly, to improve the performance of speaker-independent experiment, we propose a new phoneme stream network to estimate the articulatory parameters as the phoneme features. To the best of our knowledge, this is the first work that extracts the speaker-independent features from phonemes to improve the performance of AAI. Secondly, in order to better represent the speech information, we train a speech stream network to combine the local features and the global features. Compared with state-of-the-art (SOTA), the proposed method reduces 0.18mm on RMSE and increases 6.0% on Pearson correlation coefficient in the speaker-independent experiment. The code has been released at https://github.com/liujinyu123/AAINetwork-SPN.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑