EARTalking：端到端的GPT风格自回归说话头合成与帧级控制
EARTalking: End-to-end GPT-style Autoregressive Talking Head Synthesis with Frame-wise Control
AI总结：
本文提出EARTalking，一种端到端的GPT风格自回归模型，用于交互式音频驱动说话头生成，通过引入帧级流式生成范式和Sink Frame Window Attention机制，实现高效可控的视频生成。
摘要：
音频驱动的说话头生成旨在从静态肖像和语音生成生动逼真的视频。现有AR方法依赖中间面部表示，限制了表达性和真实性。同时，扩散方法逐片段生成，缺乏细粒度控制且存在固有延迟。为解决这些限制，我们提出EARTalking，一种新颖的端到端、GPT风格自回归模型。我们的方法引入了一种新颖的帧级、上下文驱动、音频驱动的流式生成范式。为支持可变长度视频生成和身份一致性，我们提出了Sink Frame Window Attention（SFA）机制。此外，为避免先前工作所需的复杂、分离的网络以处理多样化的控制信号，我们提出了一种流式帧条件上下文（FCIC）方案。该方案高效地在流式、上下文驱动的方式中注入多样化的控制信号，实现每帧和任意时刻的交互控制。实验表明，EARTalking优于现有自回归方法，并达到扩散方法的性能。我们的工作展示了上下文流式自回归控制的可行性，解锁了灵活、高效的生成方向。代码将被释放以供复现。
英文摘要：
Audio-driven talking head generation aims to create vivid and realistic videos from a static portrait and speech. Existing AR-based methods rely on intermediate facial representations, which limit their expressiveness and realism. Meanwhile, diffusion-based methods generate clip-by-clip, lacking fine-grained control and causing inherent latency due to overall denoising across the window. To address these limitations, we propose EARTalking, a novel end-to-end, GPT-style autoregressive model for interactive audio-driven talking head generation. Our method introduces a novel frame-by-frame, in-context, audio-driven streaming generation paradigm. For inherently supporting variable-length video generation with identity consistency, we propose the Sink Frame Window Attention (SFA) mechanism. Furthermore, to avoid the complex, separate networks that prior works required for diverse control signals, we propose a streaming Frame Condition In-Context (FCIC) scheme. This scheme efficiently injects diverse control signals in a streaming, in-context manner, enabling interactive control at every frame and at arbitrary moments. Experiments demonstrate that EARTalking outperforms existing autoregressive methods and achieves performance comparable to diffusion-based methods. Our work demonstrates the feasibility of in-context streaming autoregressive control, unlocking a scalable direction for flexible, efficient generation. The code will be released for reproducibility.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
