SoulX-FlashHead: 基于Oracle的无限实时流式谈话头生成
SoulX-FlashHead: Oracle-guided Generation of Infinite Real-time Streaming Talking Heads
AI总结：
SoulX-FlashHead通过引入流式感知预训练和Oracle引导蒸馏，实现了高保真实时流式谈话头生成，达到最先进的性能并支持超快交互。
摘要：
在音频驱动的肖像生成中，实现高保真视觉质量和低延迟流式传输之间的平衡仍是一个严峻挑战。现有大规模模型往往面临不可接受的计算成本，而轻量级替代方案通常在整体面部表示和时间稳定性上做出妥协。在本文中，我们提出了SoulX-FlashHead，一个统一的13亿参数框架，用于实时、无限长度和高保真的流式视频生成。为了解决流式场景中音频特征的不稳定性，我们引入了流式感知的时空预训练，配备了时间音频上下文缓存机制，以确保从短音频片段中提取稳健的特征。此外，为了缓解长序列自回归生成中固有的误差累积和身份漂移问题，我们提出了Oracle引导的双向蒸馏，利用真实运动先验提供精确的物理指导。我们还展示了VividHead，一个大规模高质量的数据集，包含782小时严格对齐的视频片段，以支持稳健的训练。广泛的实验表明，SoulX-FlashHead在HDTF和VFHQ基准上实现了最先进的性能。值得注意的是，我们的轻量版在单个NVIDIA RTX 4090上实现了96 FPS的推理速度，使超快交互在不牺牲视觉一致性的情况下成为可能。
英文摘要：
Achieving a balance between high-fidelity visual quality and low-latency streaming remains a formidable challenge in audio-driven portrait generation. Existing large-scale models often suffer from prohibitive computational costs, while lightweight alternatives typically compromise on holistic facial representations and temporal stability. In this paper, we propose SoulX-FlashHead, a unified 1.3B-parameter framework designed for real-time, infinite-length, and high-fidelity streaming video generation. To address the instability of audio features in streaming scenarios, we introduce Streaming-Aware Spatiotemporal Pre-training equipped with a Temporal Audio Context Cache mechanism, which ensures robust feature extraction from short audio fragments. Furthermore, to mitigate the error accumulation and identity drift inherent in long-sequence autoregressive generation, we propose Oracle-Guided Bidirectional Distillation, leveraging ground-truth motion priors to provide precise physical guidance. We also present VividHead, a large-scale, high-quality dataset containing 782 hours of strictly aligned footage to support robust training. Extensive experiments demonstrate that SoulX-FlashHead achieves state-of-the-art performance on HDTF and VFHQ benchmarks. Notably, our Lite variant achieves an inference speed of 96 FPS on a single NVIDIA RTX 4090, facilitating ultra-fast interaction without sacrificing visual coherence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
