arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2110.02891cs.LGcs.SDeess.AS
收藏

风格均衡:可控生成序列模型的无监督学习

Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models

Apple(苹果公司);University of California, San Diego(加州大学圣地亚哥分校)

Jen-Hao Rick Chang, Ashish Shrivastava, Hema Swetha Koppula, Xiaoshuai Zhang, Oncel Tuzel

摘要:

具有提取和复制特定示例风格的能力的可控生成序列模型可以实现许多应用,包括以不同声音讲述有声读物,自动完成和自动纠正手写,以及为下游识别任务生成缺失的训练样本。然而,在无监督风格设置下,用于可控序列生成模型的典型训练算法遭受训练-推理失配,其中在训练期间使用相同的样本作为内容和风格输入,但在推理期间给出不成对的样本。在本文中,我们解决了可控生成序列模型的无监督学习过程中遇到的训练-推理失配问题。所提出的方法是简单而有效的,其中我们使用一个风格转换模块,将目标风格信息转换成一个无关的风格输入。该方法使得能够使用未配对的内容和样式样本进行训练,从而减轻训练-推断不匹配。我们将风格均衡应用于三个数据集上的文本到语音和文本到手写的合成。我们进行全面的评估,包括定量和定性的用户研究。我们的研究结果表明,通过减轻训练推理与建议的风格均衡不匹配,我们实现了风格复制分数与我们的用户研究中的真实数据相当。

英文摘要:

Controllable generative sequence models with the capability to extract and replicate the style of specific examples enable many applications, including narrating audiobooks in different voices, auto-completing and auto-correcting written handwriting, and generating missing training samples for downstream recognition tasks. However, under an unsupervised-style setting, typical training algorithms for controllable sequence generative models suffer from the training-inference mismatch, where the same sample is used as content and style input during training but unpaired samples are given during inference. In this paper, we tackle the training-inference mismatch encountered during unsupervised learning of controllable generative sequence models. The proposed method is simple yet effective, where we use a style transformation module to transfer target style information into an unrelated style input. This method enables training using unpaired content and style samples and thereby mitigate the training-inference mismatch. We apply style equalization to text-to-speech and text-to-handwriting synthesis on three datasets. We conduct thorough evaluation, including both quantitative and qualitative user studies. Our results show that by mitigating the training-inference mismatch with the proposed style equalization, we achieve style replication scores comparable to real data in our user studies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑