ICE-Talk: an Interface for a Controllable Expressive Talking Machine

AnyoneNet: Synchronized Speech and Talking Head Generation for Arbitrary Person

Automatically generating videos in which synthesized speech is synchronized with lip movements in a talking head has great potential in many human-computer interaction scenarios. In this paper, we present an automatic method to generate synchronized speech and talking-head videos on the basis of text and a single face image of an arbitrary person as input. In contrast to previous text-driven talking head generation methods, which can only synthesize the voice of a specific person, the proposed method is capable of synthesizing speech for any person that is inaccessible in the training stage. S…

Improving Few-Shot Learning for Talking Face System with TTS Data Augmentation

Audio-driven talking face has attracted broad interest from academia and industry recently. However, data acquisition and labeling in audio-driven talking face are labor-intensive and costly. The lack of data resource results in poor synthesis effect. To alleviate this issue, we propose to use TTS (Text-To-Speech) for data augmentation to improve few-shot ability of the talking face system. The misalignment problem brought by the TTS audio is solved with the introduction of soft-DTW, which is first adopted in the talking face task. Moreover, features extracted by HuBERT are explored to utilize…

Ada-TTA：迈向自适应高质量的文本到说话人虚拟人合成

Ada-TTA: Towards Adaptive High-Quality Text-to-Talking Avatar Synthesis

提出Ada-TTA方法，通过零样本多说话人TTS模型解耦文本、音色和韵律，并结合神经渲染技术，在低资源条件下生成高保真、唇音同步且保持身份的说话人虚拟人视频。

双音频中心模态耦合用于说话头生成

Dual Audio-Centric Modality Coupling for Talking Head Generation

提出双音频中心模态耦合（DAMC）的NeRF框架，通过双编码器与跨同步融合模块提升说话头生成的唇形同步与图像质量，优于现有方法。

会说话的人脸：从文本联合合成说话人脸与语音

Faces that Speak: Jointly Synthesising Talking Face and Speech from Text

本研究提出统一框架，整合TFG与TTS系统，通过条件流匹配动作采样器和新型TTS条件调节方法，实现从文本同时生成自然说话人脸与匹配语音，且可泛化到未知身份。

Ditto：用于可控实时说话头合成的运动空间扩散

Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesis

Ditto通过运动空间扩散模型实现可控实时说话头合成，优化架构与训练策略以提升生成质量与实时性能。