MAVFlow：采用条件流匹配保留副语言元素的零样本AV2AV多语言翻译
MAVFlow: Preserving Paralinguistic Elements with Conditional Flow Matching for Zero-Shot AV2AV Multilingual Translation
AI总结：
针对AV2AV翻译中说话人一致性难维持的问题，提出带音视频双模态引导的条件流匹配零样本渲染器MAVFlow，通过x-vectors增强语音一致性、传递情感细节，提升了合成语音与面部生成质量。
摘要：
尽管文本转语音（TTS）模型近年来取得了进展，但音视频到音视频（AV2AV）翻译仍面临一个关键挑战：在原始与翻译后的语音及面部特征之间保持说话人一致性。为解决这一问题，我们提出了一种条件流匹配（CFM）零样本音视频渲染器，它利用来自音频和视觉模态的强双重引导。通过将多模态引导与CFM结合，我们的模型能够稳健地保留说话人专属特征，并增强零样本AV2AV翻译能力。在音频模态方面，我们通过将鲁棒说话人嵌入与x-vectors相结合来优化CFM过程，以增强说话人一致性。此外，我们将情感细微差别传递给面部渲染模块。音频和视觉线索提供的引导独立于语义或语言内容，使得我们的渲染器能够有效处理不同语言中单语说话人的零样本翻译任务。我们通过实验证明，加入以面部信息为条件的高质量梅尔频谱图，不仅提升了合成语音的质量，还对面部生成产生积极影响，从而在LSE和FID分数上实现整体性能提升。我们的代码可在https://github.com/Peter-SungwooCho/MAVFlow获取。
英文摘要：
Despite recent advances in text-to-speech (TTS) models, audio-visual-to-audio-visual (AV2AV) translation still faces a critical challenge: maintaining speaker consistency between the original and translated vocal and facial features. To address this issue, we propose a conditional flow matching (CFM) zero-shot audio-visual renderer that utilizes strong dual guidance from both audio and visual modalities. By leveraging multimodal guidance with CFM, our model robustly preserves speaker-specific characteristics and enhances zero-shot AV2AV translation abilities. For the audio modality, we enhance the CFM process by integrating robust speaker embeddings with x-vectors, which serve to bolster speaker consistency. Additionally, we convey emotional nuances to the face rendering module. The guidance provided by both audio and visual cues remains independent of semantic or linguistic content, allowing our renderer to effectively handle zero-shot translation tasks for monolingual speakers in different languages. We empirically demonstrate that the inclusion of high-quality mel-spectrograms conditioned on facial information not only enhances the quality of the synthesized speech but also positively influences facial generation, leading to overall performance improvements in LSE and FID score. Our code is available at https://github.com/Peter-SungwooCho/MAVFlow.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
