UniSpeaker：一种面向多模态驱动说话人生成的统一方法
Unispeaker: A Unified Approach for Multimodality-driven Speaker Generation
AI总结：
针对多模态说话人生成的发展需求，本文提出基于KV-Former统一语音聚合器与软对比损失的UniSpeaker方法，构建首个MVC基准，实验显示其在五项任务上优于现有单模态模型。
摘要：
个性化语音生成的最新进展已让合成语音愈发接近目标说话人录音的真实感，但多模态说话人生成仍处于发展阶段。本文提出UniSpeaker，一种面向多模态驱动说话人生成的统一方法。具体而言，我们提出基于KV-Former的统一语音聚合器，采用软对比损失将多样的语音描述模态映射至共享语音空间，确保生成的语音与输入描述更契合。为评估多模态驱动的语音控制，我们构建了首个基于多模态的语音控制（MVC）基准，聚焦语音适配性、语音多样性与语音质量。我们使用MVC基准在五项任务上对UniSpeaker进行评估，实验结果表明UniSpeaker性能优于以往的单模态特定模型。语音样本可访问https://UniSpeaker.github.io获取。
英文摘要：
Recent advancements in personalized speech generation have brought synthetic speech increasingly close to the realism of target speakers' recordings, yet multimodal speaker generation remains on the rise. This paper introduces UniSpeaker, a unified approach for multimodality-driven speaker generation. Specifically, we propose a unified voice aggregator based on KV-Former, applying soft contrastive loss to map diverse voice description modalities into a shared voice space, ensuring that the generated voice aligns more closely with the input descriptions. To evaluate multimodality-driven voice control, we build the first multimodality-based voice control (MVC) benchmark, focusing on voice suitability, voice diversity, and speech quality. UniSpeaker is evaluated across five tasks using the MVC benchmark, and the experimental results demonstrate that UniSpeaker outperforms previous modality-specific models. Speech samples are available at \url{https://UniSpeaker.github.io}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
