arXiv 2311.002412024-02-01cs.CV

1DFormer：一种用于人脸关键点跟踪的学习1D关键点表征的Transformer架构

1DFormer: a Transformer Architecture Learning 1D Landmark Representations for Facial Landmark Tracking

iFLYTEK Research（讯飞研究院）；University of Science and Technology of China（中国科学技术大学）

Shi Yin, Shijie Huan, Shangfei Wang, Jinshui Hu, Tao Guo, Bing Yin, Baocai Yin, Cong Liu

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对现有方法未充分挖掘1D关键点表征时序与结构建模潜力的问题，提出1DFormer架构，通过时序与空间维度的令牌交互学习表征，在人脸关键点跟踪任务上达到SOTA性能。

摘要：

近年来，基于1D关键点表征的热图回归方法在人脸关键点定位任务中展现出优异性能。然而，现有方法未能深入挖掘1D关键点表征在多关键点时序与结构建模方面的潜力，以服务于人脸关键点跟踪任务。针对这一局限，我们提出一种名为1DFormer的Transformer架构，它通过在时间与空间维度上进行令牌间的信息交互，捕捉关键点的动态变化与几何模式，从而学习具备丰富信息的1D关键点表征，用于人脸关键点跟踪。在时序建模方面，我们提出循环令牌混合机制、轴-关键点-位置嵌入机制，以及置信度增强的多头注意力机制，能够自适应且鲁棒地将长期关键点动态信息嵌入其1D表征中；在结构建模方面，我们设计了组内与组间结构建模机制，通过1D卷积层在空间维度上进行令牌交互，编码组件级与全局级的人脸结构模式，以此优化关键点的1D表征。在300VW和TF数据集上的实验结果表明，1DFormer能够成功建模长程序列模式以及人脸固有结构，学习到信息丰富的关键点序列1D表征，并且在人脸关键点跟踪任务上取得了当前最优性能。

英文摘要：

Recently, heatmap regression methods based on 1D landmark representations have shown prominent performance on locating facial landmarks. However, previous methods ignored to make deep explorations on the good potentials of 1D landmark representations for sequential and structural modeling of multiple landmarks to track facial landmarks. To address this limitation, we propose a Transformer architecture, namely 1DFormer, which learns informative 1D landmark representations by capturing the dynamic and the geometric patterns of landmarks via token communications in both temporal and spatial dimensions for facial landmark tracking. For temporal modeling, we propose a recurrent token mixing mechanism, an axis-landmark-positional embedding mechanism, as well as a confidence-enhanced multi-head attention mechanism to adaptively and robustly embed long-term landmark dynamics into their 1D representations; for structure modeling, we design intra-group and inter-group structure modeling mechanisms to encode the component-level as well as global-level facial structure patterns as a refinement for the 1D representations of landmarks through token communications in the spatial dimension via 1D convolutional layers. Experimental results on the 300VW and the TF databases show that 1DFormer successfully models the long-range sequential patterns as well as the inherent facial structures to learn informative 1D representations of landmark sequences, and achieves state-of-the-art performance on facial landmark tracking.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。