基于掩码选择性状态空间建模的音视频可控视频扩散用于自然说话头生成
Audio-visual Controlled Video Diffusion with Masked Selective State Spaces Modeling for Natural Talking Head Generation
AI总结：
本文提出ACTalker视频扩散框架，通过并行mamba结构和掩码丢弃策略实现多信号独立控制面部区域，生成自然协调的说话头视频。
摘要：
说话头合成对于虚拟化身和人机交互至关重要。然而，大多数现有方法通常仅限于接受单一主模态的控制，限制了其实际应用价值。为此，我们提出了ACTalker，这是一个端到端的视频扩散框架，支持多信号控制和单信号控制的说话头视频生成。对于多重控制，我们设计了一种具有多个分支的并行mamba结构，每个分支利用独立的驱动信号来控制特定的面部区域。在所有分支之间应用门控机制，为视频生成提供灵活的控制。为了确保受控视频在时间和空间上的自然协调，我们采用了mamba结构，使驱动信号能够在每个分支中跨两个维度操纵特征令牌。此外，我们引入了一种掩码丢弃策略，允许每个驱动信号在mamba结构内独立控制其对应的面部区域，防止控制冲突。实验结果表明，我们的方法能够生成由多样化信号驱动的自然面部视频，并且mamba层能够无缝整合多种驱动模态而不产生冲突。项目网站可在https://harlanhong.github.io/publications/actalker/index.html找到。
英文摘要：
Talking head synthesis is vital for virtual avatars and human-computer interaction. However, most existing methods are typically limited to accepting control from a single primary modality, restricting their practical utility. To this end, we introduce \textbf{ACTalker}, an end-to-end video diffusion framework that supports both multi-signals control and single-signal control for talking head video generation. For multiple control, we design a parallel mamba structure with multiple branches, each utilizing a separate driving signal to control specific facial regions. A gate mechanism is applied across all branches, providing flexible control over video generation. To ensure natural coordination of the controlled video both temporally and spatially, we employ the mamba structure, which enables driving signals to manipulate feature tokens across both dimensions in each branch. Additionally, we introduce a mask-drop strategy that allows each driving signal to independently control its corresponding facial region within the mamba structure, preventing control conflicts. Experimental results demonstrate that our method produces natural-looking facial videos driven by diverse signals and that the mamba layer seamlessly integrates multiple driving modalities without conflict. The project website can be found at https://harlanhong.github.io/publications/actalker/index.html.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
