PhysFormer ++：基于SlowFast时间差分Transformer的面部视频生理测量
PhysFormer++: Facial Video-based Physiological Measurement with SlowFast Temporal Difference Transformer
摘要：
远程光电体积描记术（rPPG）的目的是在没有任何接触的情况下从面部视频测量心脏活动和生理信号，其在许多应用中具有巨大的潜力（例如，远程保健和情感计算）。最近的深度学习方法专注于使用具有有限时空感受野的卷积神经网络来挖掘细微的rPPG线索，其忽略了用于rPPG建模的长距离时空感知和交互。在本文中，我们提出了两个端到端的视频Transformer为基础的架构，即PhysFormer和PhysFormer++，自适应聚合本地和全球的时空特征rPPG表示增强。作为PhysFormer中的关键模块，时域差分Transformers首先利用时域差分引导的全局注意力增强准周期rPPG特征，然后针对干扰改进局部时空表示。为了更好地利用时间上下文和周期性rPPG线索，我们还扩展了PhysFormer到基于双通道SlowFast的PhysFormer++，具有时间差周期性和交叉注意Transformers。此外，本文还提出了频域标签分布学习和课程学习启发的动态约束机制，为PhysFormer和PhysFormer++提供了精细的监督机制，减轻了过拟合现象。在四个基准数据集上进行了全面的实验，以显示我们在数据集内和跨数据集测试上的优越性能。与大多数需要从大规模数据集进行预训练的Transformer网络不同，所提出的PhysFormer系列可以在rPPG数据集上从头开始轻松训练，这使得它有望成为rPPG社区的新型Transformer基线。
英文摘要：
Remote photoplethysmography (rPPG), which aims at measuring heart activities and physiological signals from facial video without any contact, has great potential in many applications (e.g., remote healthcare and affective computing). Recent deep learning approaches focus on mining subtle rPPG clues using convolutional neural networks with limited spatio-temporal receptive fields, which neglect the long-range spatio-temporal perception and interaction for rPPG modeling. In this paper, we propose two end-to-end video transformer based architectures, namely PhysFormer and PhysFormer++, to adaptively aggregate both local and global spatio-temporal features for rPPG representation enhancement. As key modules in PhysFormer, the temporal difference transformers first enhance the quasi-periodic rPPG features with temporal difference guided global attention, and then refine the local spatio-temporal representation against interference. To better exploit the temporal contextual and periodic rPPG clues, we also extend the PhysFormer to the two-pathway SlowFast based PhysFormer++ with temporal difference periodic and cross-attention transformers. Furthermore, we propose the label distribution learning and a curriculum learning inspired dynamic constraint in frequency domain, which provide elaborate supervisions for PhysFormer and PhysFormer++ and alleviate overfitting. Comprehensive experiments are performed on four benchmark datasets to show our superior performance on both intra- and cross-dataset testings. Unlike most transformer networks needed pretraining from large-scale datasets, the proposed PhysFormer family can be easily trained from scratch on rPPG datasets, which makes it promising as a novel transformer baseline for the rPPG community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
