OSM-Net：具有自然头部动作的一对多单样本说话人头像生成
OSM-Net: One-to-Many One-shot Talking Head Generation with Spontaneous Head Motions
AI总结：
针对单样本说话人头像中头部动作缺失或一对一映射不自然的问题，本文提出 OSM-Net，通过片段级运动空间与音频条件采样实现一对多自然头部动作生成。
摘要：
单样本说话人头像生成没有显式的头部运动参考，因此难以生成带有头部动作的说话人头像。一些现有工作仅编辑嘴部区域并生成静止的说话人头像，导致说话人头像表现不真实。另一些工作在音频信号与头部运动序列之间构建一对一映射，这会在映射中引入歧义对应关系，因为人们在说相同内容时可能表现出不同的头部动作。这种不合理的映射形式无法对多样性建模，会产生近乎静止甚至夸张的头部动作，既不自然又怪异。因此，单样本说话人头像生成任务实际上是一个一对多的不适定问题，人们在说话时会呈现多样的头部动作。基于上述观察，我们提出 OSM-Net，这是一个具有自然头部动作的一对多单样本说话人头像生成网络。OSM-Net 构建了一个运动空间，其中包含丰富多样的片段级头部运动特征。该空间的每个基表示一个片段中有意义的头部运动特征，而不仅仅是单帧特征，从而在说话人头像中提供更连贯、更自然的运动变化。驱动音频被映射到该运动空间，可以在其周围合理范围内采样各种运动特征，以实现一对多映射。此外，关键点约束和时间窗口特征输入改善了准确的表情特征提取与视频生成。大量实验表明，与其他方法相比，OSM-Net 在合理的一对多映射范式下生成了更自然、更真实的头部动作。
英文摘要：
One-shot talking head generation has no explicit head movement reference, thus it is difficult to generate talking heads with head motions. Some existing works only edit the mouth area and generate still talking heads, leading to unreal talking head performance. Other works construct one-to-one mapping between audio signal and head motion sequences, introducing ambiguity correspondences into the mapping since people can behave differently in head motions when speaking the same content. This unreasonable mapping form fails to model the diversity and produces either nearly static or even exaggerated head motions, which are unnatural and strange. Therefore, the one-shot talking head generation task is actually a one-to-many ill-posed problem and people present diverse head motions when speaking. Based on the above observation, we propose OSM-Net, a \textit{one-to-many} one-shot talking head generation network with natural head motions. OSM-Net constructs a motion space that contains rich and various clip-level head motion features. Each basis of the space represents a feature of meaningful head motion in a clip rather than just a frame, thus providing more coherent and natural motion changes in talking heads. The driving audio is mapped into the motion space, around which various motion features can be sampled within a reasonable range to achieve the one-to-many mapping. Besides, the landmark constraint and time window feature input improve the accurate expression feature extraction and video generation. Extensive experiments show that OSM-Net generates more natural realistic head motions under reasonable one-to-many mapping paradigm compared with other methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
