使用基于音素计数比例奖励的强化学习的等长神经机器翻译
Isometric Neural Machine Translation using Phoneme Count Ratio Reward-based Reinforcement Learning
AI总结：
本文提出一种基于强化学习的等长NMT系统，通过优化源语言与目标语言间的音素计数对齐来保证视频配音音画同步，并引入PCC分数评估，在英语-印地语对上较现有模型提升约36%，同时利用师生架构平衡音素计数与翻译质量。
摘要：
传统的自动视频配音（AVD）流水线由三个关键模块组成，即自动语音识别（ASR）、神经机器翻译（NMT）和文本转语音（TTS）。在AVD流水线中，采用等长NMT算法来调节合成输出文本的长度。这是为了保证在配音过程之后视频和音频的对齐同步。以前的方法主要集中在机器翻译模型源语言和目标语言文本中字符和单词数量的对齐上。然而，我们的方法旨在转而对齐音素的数量，因为它们与语音持续时间密切相关。在本文中，我们提出了使用强化学习（RL）开发等长NMT系统，重点优化源语言和目标语言句子对中音素计数的对齐。为了评估我们的模型，我们提出了音素计数合规性（PCC）分数，这是一种长度合规性的度量。当应用于英语-印地语语言对时，与最先进的模型相比，我们的方法在PCC分数上显示出约36%的显著提升。此外，我们在RL方法的框架内提出了一种师生架构，以在音素计数和翻译质量之间保持权衡。
英文摘要：
Traditional Automatic Video Dubbing (AVD) pipeline consists of three key modules, namely, Automatic Speech Recognition (ASR), Neural Machine Translation (NMT), and Text-to-Speech (TTS). Within AVD pipelines, isometric-NMT algorithms are employed to regulate the length of the synthesized output text. This is done to guarantee synchronization with respect to the alignment of video and audio subsequent to the dubbing process. Previous approaches have focused on aligning the number of characters and words in the source and target language texts of Machine Translation models. However, our approach aims to align the number of phonemes instead, as they are closely associated with speech duration. In this paper, we present the development of an isometric NMT system using Reinforcement Learning (RL), with a focus on optimizing the alignment of phoneme counts in the source and target language sentence pairs. To evaluate our models, we propose the Phoneme Count Compliance (PCC) score, which is a measure of length compliance. Our approach demonstrates a substantial improvement of approximately 36% in the PCC score compared to the state-of-the-art models when applied to English-Hindi language pairs. Moreover, we propose a student-teacher architecture within the framework of our RL approach to maintain a trade-off between the phoneme count and translation quality.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
