STARCaster：用于身份和视图感知的会说话肖像的时空自回归视频扩散
STARCaster: Spatio-Temporal AutoRegressive Video Diffusion for Identity- and View-Aware Talking Portraits
AI总结：
研究提出STARCaster模型，通过重新思考参考和几何范式，采用组合方法及解耦学习，解决语音驱动肖像动画和动态视点控制问题，能有效泛化，超越先前方法。
摘要：
本文提出了STARCaster，一种身份感知的时空视频扩散模型，在统一框架内，给定身份嵌入或参考图像，解决语音驱动的肖像动画和动态视点控制问题。现有2D语音到视频扩散模型严重依赖参考指导，导致运动多样性有限，3D感知动画通常依赖预训练三平面生成器的反演，常导致重建不完美和身份漂移。我们从两方面重新思考基于参考和几何的范式，一是通过引入更软的身份约束偏离预训练时的严格参考条件，二是通过利用视频数据固有的多视图性质在2D视频域内隐式解决3D感知问题。STARCaster采用组合方法，从身份感知运动建模，到基于唇读监督的视听同步，最后通过时间到空间的适配实现新颖视图动画。为克服4D视听数据稀缺问题，提出解耦学习方法，独立训练视图一致性和时间连贯性。综合评估表明，STARCaster在不同任务和身份上有效泛化，始终超越先前方法。
英文摘要：
This paper presents STARCaster, an identity-aware spatio-temporal video diffusion model that addresses both speech-driven portrait animation and dynamic viewpoint control, given an identity embedding or reference image, within a unified framework. Existing 2D speech-to-video diffusion models depend heavily on reference guidance, leading to limited motion diversity. At the same time, 3D-aware animation typically relies on inversion through pretrained tri-plane generators, which often leads to imperfect reconstructions and identity drift. We rethink reference- and geometry-based paradigms in two ways. First, we deviate from strict reference conditioning at pretraining by introducing softer identity constraints. Second, we address 3D awareness implicitly within the 2D video domain by leveraging the inherent multi-view nature of video data. STARCaster adopts a compositional approach progressing from ID-aware motion modeling, to audio-visual synchronization via lip reading-based supervision, and finally to novel view animation through temporal-to-spatial adaptation. To overcome the scarcity of 4D audio-visual data, we propose a decoupled learning approach in which view consistency and temporal coherence are trained independently. Comprehensive evaluations demonstrate that STARCaster generalizes effectively across tasks and identities, consistently surpassing prior approaches in different benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
