面向持续自我中心活动识别：一个用于持续学习的多模态自我中心活动数据集
Towards Continual Egocentric Activity Recognition: A Multi-modal Egocentric Activity Dataset for Continual Learning
AI总结：
针对自我中心活动识别在多模态持续学习中的灾难性遗忘问题，本文提出了多模态数据集UESTC-MMEA-CL，并基于RGB、加速度和陀螺仪三种模态对四种基线方法进行了广泛评估。
摘要：
随着可穿戴相机的快速发展，用于第一人称视觉感知的自我中心视频大量涌现。利用自我中心视频预测第一人称活动面临诸多挑战，包括视野受限、遮挡和运动不稳定。观察到可穿戴设备的传感器数据有助于人类活动识别，多模态活动识别正受到越来越多的关注。然而，相关数据集的缺失阻碍了用于自我中心活动识别的多模态深度学习的发展。如今，现实世界中的深度学习已将焦点转向持续学习，而该学习常遭受灾难性遗忘。但由于缺乏数据集，自我中心活动识别中的灾难性遗忘问题，尤其是在多模态背景下，仍未被探索。为推动此研究，我们提出了一个用于持续学习的多模态自我中心活动数据集，名为UESTC-MMEA-CL，该数据集由集成了第一人称相机和可穿戴传感器的自研眼镜采集。它包含视频、加速度计和陀螺仪的同步数据，涵盖10名参与者执行的32种日常活动。其类别类型和规模与其他公开数据集进行了对比。给出了传感器数据的统计分析，以展示对不同行为的辅助效果。在基础网络架构上，报告了分别和联合使用RGB、加速度和陀螺仪三种模态时的自我中心活动识别结果。为探索持续学习任务中的灾难性遗忘，使用不同的多模态组合对四种基线方法进行了广泛评估。我们希望UESTC-MMEA-CL能促进可穿戴应用中第一人称活动识别持续学习的未来研究。
英文摘要：
With the rapid development of wearable cameras, a massive collection of egocentric video for first-person visual perception becomes available. Using egocentric videos to predict first-person activity faces many challenges, including limited field of view, occlusions, and unstable motions. Observing that sensor data from wearable devices facilitates human activity recognition, multi-modal activity recognition is attracting increasing attention. However, the deficiency of related dataset hinders the development of multi-modal deep learning for egocentric activity recognition. Nowadays, deep learning in real world has led to a focus on continual learning that often suffers from catastrophic forgetting. But the catastrophic forgetting problem for egocentric activity recognition, especially in the context of multiple modalities, remains unexplored due to unavailability of dataset. In order to assist this research, we present a multi-modal egocentric activity dataset for continual learning named UESTC-MMEA-CL, which is collected by self-developed glasses integrating a first-person camera and wearable sensors. It contains synchronized data of videos, accelerometers, and gyroscopes, for 32 types of daily activities, performed by 10 participants. Its class types and scale are compared with other publicly available datasets. The statistical analysis of the sensor data is given to show the auxiliary effects for different behaviors. And results of egocentric activity recognition are reported when using separately, and jointly, three modalities: RGB, acceleration, and gyroscope, on a base network architecture. To explore the catastrophic forgetting in continual learning tasks, four baseline methods are extensively evaluated with different multi-modal combinations. We hope the UESTC-MMEA-CL can promote future studies on continual learning for first-person activity recognition in wearable applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
