音频驱动说话头生成的时序对齐评估
Temporally-Aligned Evaluation for Audio-Driven Talking Head Generation
AI总结：
针对现有帧级评估指标对时序偏差敏感的问题，提出基于软动态时间规整的序列级对齐评估框架，提升评估鲁棒性并揭示不同建模范式间的系统权衡。
摘要：
音频驱动的说话头生成技术发展迅速，但现有评估协议主要依赖帧级指标，假设生成视频与参考视频之间存在严格的时间对应关系。这一假设与语音驱动的面部运动不符，后者自然包含轻微的时间偏移、不同的说话速度和风格变化。因此，传统指标可能将无害的时间差异视为质量错误，使得公平比较方法并理解其权衡变得更加困难。在这项工作中，我们认为动态生成模型的评估应被表述为序列对齐问题，而非独立的帧比较。我们引入了一种统一的序列级重新表述，将软动态时间规整集成到已有的评估流程中。通过在对齐特征轨迹的同时保持时间顺序，所提出的框架对有限的时间错位具有鲁棒性，且不改变底层的感知、身份或同步编码器。我们表明，在刚性对齐下，帧级评估可被视为一个特例，而序列级对齐提供了更好的稳定性、对时间差异的更低敏感性以及建模范式之间更清晰的区分。基于这一原则性表述，我们在标准化协议下，对涵盖规范、野外和风格多样场景的七个数据集上的20种方法进行了大规模基准测试。大量实验表明，时序对齐的指标对时间差异更鲁棒，跨数据集提供更一致的结果，并能更好地揭示建模范式之间的系统权衡，例如同步性与真实性、表现力与稳定性之间的权衡。
英文摘要：
Audio-driven talking-head generation has advanced rapidly, yet existing evaluation protocols mainly rely on frame-wise metrics that assume strict temporal correspondence between generated and reference videos. This assumption does not match speech-driven facial motion, which naturally includes slight timing shifts, different speaking speeds, and stylistic variations. As a result, conventional metrics may treat harmless timing differences as quality errors, making it harder to fairly compare methods and understand their trade-offs. In this work, we argue that evaluation of dynamic generative models should be formulated as a sequence-alignment problem rather than independent frame comparison. We introduce a unified sequence-level reformulation that integrates Soft Dynamic Time Warping into established evaluation pipelines. By aligning feature trajectories while preserving temporal order, the proposed framework provides robustness to bounded temporal misalignments without altering the underlying perceptual, identity, or synchronization encoders. We show that frame-wise evaluation can be viewed as a special case under rigid alignment, while sequence-level alignment provides improved stability, lower sensitivity to timing differences, and clearer separation between modeling paradigms. Building on this principled formulation, we conduct a large-scale benchmark of 20 methods across seven datasets spanning canonical, in-the-wild, and style-diverse scenarios under standardized protocols. Extensive experiments show that temporally aligned metrics are more robust to timing differences, provide more consistent results across datasets, and better reveal systematic trade-offs between modeling paradigms, such as synchronization versus realism and expressiveness versus stability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
