自我中心视频嵌入编码的内容及其与惯性传感器结合用于人类活动识别的方法
What Egocentric Video Embeddings Encode and How to Combine Them with Inertial Sensors for Human Activity Recognition
AI总结：
该研究探究自我中心视频嵌入的编码内容，发现其主导方向由摄像头朝向决定，结合惯性信号在WEAR数据集上通过视频图传播惯性预测，将宏F1从0.648提升至0.852，为人类活动识别提供了新方法。
摘要：
自我中心视频与佩戴在身体上的惯性传感器正被越来越多地同步记录，且视频通常由预训练模型进行编码。当摄像头佩戴在头部时，这些嵌入所代表的内容，以及应如何将其与惯性传感数据相结合，目前受到的关注较少。针对VideoMAEv2、CLIP、DINOv2和自我中心VC-1模型，嵌入的主导主方向主要由摄像头的朝向决定。在Ego-Exo4D数据集上，测量得到的头部朝向可解释自我中心摄像头（R²为0.49-0.70）的嵌入主方向，但无法解释异中心摄像头的；而在WEAR数据集上，场景布局可解释这些嵌入的主方向。嵌入中可被肢体惯性信号预测的部分用于区分不同人的活动，剩余部分则用于区分同一参与者的不同锻炼组，但无法区分不同人的活动。这与两者的分工一致：视频用于识别同一组的片段，惯性分类器对这些片段进行标注。在WEAR数据集上，去除时间顺序并为每一秒的片段随机选择一个肢体后，我们在去除主导成分的参与者内部视频图上传播惯性预测，这使得按受试者划分的交叉验证宏F1值从0.648提升至0.852。在我们的对比实验中，传播仅在视频图上有效，而将两种模态融合到一个分类器中，或向图中添加惯性边的表现则更差。
英文摘要：
Egocentric video and body-worn inertial sensors are increasingly recorded together, and the video is usually encoded by pretrained models. What these embeddings represent when the camera sits on the head, and how they should be combined with inertial sensing, has received little attention. Across VideoMAEv2, CLIP, DINOv2 and the egocentric VC-1, the leading principal directions of the embedding are dominated by where the camera looks. Measured head orientation explains them on Ego-Exo4D for egocentric (R^2 0.49-0.70) but not exocentric cameras, and the scene layout explains them on WEAR. The part that the limbs' inertial signals predict separates activities across people. In a set-level probe, the rest separates exercise sets within a participant but not activities across people. This is consistent with separate roles: the video finds the clips of the same set, and the inertial classifier labels them. On WEAR, with temporal order removed and one random limb per one-second clip, we propagate inertial predictions over a within-participant video graph whose leading components are removed. This raises subject-wise cross-validated macro-F1 from 0.648 to 0.852. In our comparisons, propagation helps only over the video graph, and fusing the modalities in one classifier, or adding inertial edges to the graph, performs worse.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
