Jointly Optimizing Translations and Speech Timing to Improve Isochrony in Automatic Dubbing

Isochrony-Aware Neural Machine Translation for Automatic Dubbing

We introduce the task of isochrony-aware machine translation which aims at generating translations suitable for dubbing. Dubbing of a spoken sentence requires transferring the content as well as the speech-pause structure of the source into the target language to achieve audiovisual coherence. Practically, this implies correctly projecting pauses from the source to the target and ensuring that target speech segments have roughly the same duration of the corresponding source speech segments. In this work, we propose implicit and explicit modeling approaches to integrate isochrony information in…

Prosodic Alignment for off-screen automatic dubbing

The goal of automatic dubbing is to perform speech-to-speech translation while achieving audiovisual coherence. This entails isochrony, i.e., translating the original speech by also matching its prosodic structure into phrases and pauses, especially when the speaker's mouth is visible. In previous work, we introduced a prosodic alignment model to address isochrone or on-screen dubbing. In this work, we extend the prosodic alignment model to also address off-screen dubbing that requires less stringent synchronization constraints. We conduct experiments on four dubbing directions - English to Fr…

TransVIP：具备语音与等时性保留能力的语音到语音翻译系统

TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation

针对端到端语音到语音翻译性能弱于级联模型、数据稀缺的问题，提出TransVIP框架，以级联用数据集+联合概率实现端到端推理，搭配双编码器保留音色与等时性，法英翻译效果超越现有SOTA模型。

使用目标因子和辅助计数器改进等时机器翻译

Improving Isochronous Machine Translation with Target Factors and Auxiliary Counters

针对自动配音中的等时语音翻译问题，该研究在 Transformer 模型中引入目标因子和辅助计数器，联合预测时长与音素序列并跟踪时间信息，有效提升了翻译质量与等时性。

等时性控制的语音到文本翻译：一项从汉藏语系到印欧语系翻译的研究

Isochrony-Controlled Speech-to-Text Translation: A study on translating from Sino-Tibetan to Indo-European Languages

本文提出一种等时性控制的端到端语音翻译方法，通过向解码器提供时间信息以预测语音和停顿时长，实现翻译时长与源音频的严格对齐，在CoVoST 2上取得了高语音重叠率且BLEU仅微降。

ANIM-400K：用于视频自动化端到端配音的大规模数据集

ANIM-400K: A Large-Scale Dataset for Automated End-To-End Dubbing of Video

本文提出ANIM-400K，一个包含超42.5万日英对齐动画片段的大规模数据集，以解决视频自动化端到端配音的数据匮乏问题，支持配音、同传、摘要和分类等任务。