egoPPG：利用眼动追踪摄像头在自我中心系统中进行心率估计以惠及下游视觉任务
egoPPG: Heart Rate Estimation from Eye-Tracking Cameras in Egocentric Systems to Benefit Downstream Vision Tasks
AI总结：
提出egoPPG任务和PulseFormer方法，利用眼动摄像头和惯性测量单元估计心率，在EgoExo4D上提升熟练度估计14%，并发布代码、数据集和增强。
摘要：
自我中心视觉系统旨在理解空间环境以及佩戴者在其内部的行为，包括动作、活动和交互。我们认为，自我中心系统还必须检测生理状态，以捕捉人的注意力和情境反应，这对于上下文感知的行为建模至关重要。在本文中，我们提出了egoPPG，这是一个新颖的自我中心系统视觉任务，用于恢复人的心脏活动以辅助下游视觉任务。我们引入了PulseFormer，一种从未经修改的自我中心视觉系统上的眼动追踪摄像头提取心率作为生理状态关键指标的方法。PulseFormer持续从眼睛周围区域估计光电容积脉搏波（PPG），并融合来自头戴设备惯性测量单元的运动线索以跟踪心率值。我们在EgoExo4D（一个现有的自我中心数据集）上展示了egoPPG对关键任务的下游益处，发现PulseFormer的心率估计将熟练度估计提高了14%。为了训练和验证PulseFormer，我们收集了一个包含超过13小时来自Project Aria的眼动追踪视频以及接触式PPG信号和心电图（ECG）作为真实心率值的数据集。与EgoExo4D类似，25名参与者执行了各种日常活动，如办公室工作、烹饪、跳舞和锻炼，这些活动引起了显著的自然运动和心率变化（44-164次/分钟）。我们的模型稳健地估计心率（平均绝对误差=7.67次/分钟）并捕捉模式（相关系数r=0.85）。我们的结果表明，自我中心系统如何统一环境和生理追踪以更好地理解用户，以及egoPPG作为补充任务为现有数据集和任务提供了有意义的增强。我们发布了我们的代码、数据集以及针对EgoExo4D的心率增强，以激发对生理感知自我中心任务的研究。
英文摘要：
Egocentric vision systems aim to understand the spatial surroundings and the wearer's behavior inside it, including motions, activities, and interactions. We argue that egocentric systems must additionally detect physiological states to capture a person's attention and situational responses, which are critical for context-aware behavior modeling. In this paper, we propose egoPPG, a novel vision task for egocentric systems to recover a person's cardiac activity to aid downstream vision tasks. We introduce PulseFormer, a method to extract heart rate as a key indicator of physiological state from the eye tracking cameras on unmodified egocentric vision systems. PulseFormer continuously estimates the photoplethysmogram (PPG) from areas around the eyes and fuses motion cues from the headset's inertial measurement unit to track HR values. We demonstrate egoPPG's downstream benefit for a key task on EgoExo4D, an existing egocentric dataset for which we find PulseFormer's estimates of HR to improve proficiency estimation by 14%. To train and validate PulseFormer, we collected a dataset of 13+ hours of eye tracking videos from Project Aria and contact-based PPG signals as well as an electrocardiogram (ECG) for ground-truth HR values. Similar to EgoExo4D, 25 participants performed diverse everyday activities such as office work, cooking, dancing, and exercising, which induced significant natural motion and HR variation (44-164 bpm). Our model robustly estimates HR (MAE=7.67 bpm) and captures patterns (r=0.85). Our results show how egocentric systems may unify environmental and physiological tracking to better understand users and that egoPPG as a complementary task provides meaningful augmentations for existing datasets and tasks. We release our code, dataset, and HR augmentations for EgoExo4D to inspire research on physiology-aware egocentric tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
