arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.04798cs.CV
收藏

Spiking-PhysFormer：基于并行脉冲驱动Transformer的摄像头远程光电容积脉搏波描记法

Spiking-PhysFormer: Camera-Based Remote Photoplethysmography with Parallel Spike-driven Transformer

Tsinghua University（清华大学）；Beijing Smartchip Microelectronics Technology Co., Ltd（北京智芯微电子科技有限公司）

Mingxuan Liu, Jiankai Tang, Yongli Chen, Haoxiang Li, Jiahao Qi, Siwei Li, Kegang Wang, Jie Gan, Yuntao Wang, Hong Chen

备注：
Mingxuan Liu and Jiankai Tang are co-first authors of the article. Accepted by Neural Networks
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文首次将脉冲神经网络引入远程光电容积脉搏波描记法，提出Spiking-PhysFormer混合模型，通过并行脉冲transformer块和简化脉冲自注意力机制，在保持性能的同时显著降低功耗。

摘要：

人工神经网络（ANN）可以帮助基于摄像头的远程光电容积脉搏波描记法（rPPG）更准确地从面部视频中测量心脏活动和生理信号，如脉搏波、心率和呼吸率。然而，大多数现有的基于ANN的方法需要大量的计算资源，这给在移动设备上的有效部署带来了挑战。另一方面，脉冲神经网络（SNN）由于其二元和事件驱动的架构，在节能深度学习方面具有巨大潜力。据我们所知，我们是首个将SNN引入rPPG领域的研究，提出了一个混合神经网络（HNN）模型，即Spiking-PhysFormer，旨在降低功耗。具体而言，所提出的Spiking-PhysFormer由一个基于ANN的patch embedding块、基于SNN的transformer块和一个基于ANN的预测头组成。首先，为了在保持其聚合局部和全局时空特征能力的同时简化transformer块，我们设计了一个并行脉冲transformer块来替代顺序子块。此外，我们提出了一种简化的脉冲自注意力机制，在不影响模型性能的情况下省略了value参数。在四个数据集——PURE、UBFC-rPPG、UBFC-Phys和MMPD上进行的实验表明，与PhysFormer相比，所提出的模型实现了12.4%的功耗降低。此外，transformer块的功耗降低了12.2倍，同时保持了与PhysFormer和其他基于ANN的模型相当的性能。

英文摘要：

Artificial neural networks (ANNs) can help camera-based remote photoplethysmography (rPPG) in measuring cardiac activity and physiological signals from facial videos, such as pulse wave, heart rate and respiration rate with better accuracy. However, most existing ANN-based methods require substantial computing resources, which poses challenges for effective deployment on mobile devices. Spiking neural networks (SNNs), on the other hand, hold immense potential for energy-efficient deep learning owing to their binary and event-driven architecture. To the best of our knowledge, we are the first to introduce SNNs into the realm of rPPG, proposing a hybrid neural network (HNN) model, the Spiking-PhysFormer, aimed at reducing power consumption. Specifically, the proposed Spiking-PhyFormer consists of an ANN-based patch embedding block, SNN-based transformer blocks, and an ANN-based predictor head. First, to simplify the transformer block while preserving its capacity to aggregate local and global spatio-temporal features, we design a parallel spike transformer block to replace sequential sub-blocks. Additionally, we propose a simplified spiking self-attention mechanism that omits the value parameter without compromising the model's performance. Experiments conducted on four datasets-PURE, UBFC-rPPG, UBFC-Phys, and MMPD demonstrate that the proposed model achieves a 12.4\% reduction in power consumption compared to PhysFormer. Additionally, the power consumption of the transformer block is reduced by a factor of 12.2, while maintaining decent performance as PhysFormer and other ANN-based models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑