MAND: 面向开放世界自我中心活动识别的模态感知新颖性检测
MAND: Modality-Aware Novelty Detection for Open-World Egocentric Activity Recognition
AI总结：
提出MAND框架，通过模态感知自适应评分和表示稳定训练，利用视觉和惯性模态互补信息，提升开放世界自我中心活动识别中的新颖性检测和已知类准确率。
摘要：
多模态自我中心活动识别整合视觉和惯性线索以实现鲁棒的第一人称行为理解。然而，在开放世界环境中部署此类系统需要检测新颖活动，同时从非平稳数据流中持续学习。现有方法依赖主融合logits进行新颖性评分，未充分利用各模态可用的互补证据。由于这些logits常被RGB主导，其他模态（尤其是IMU）的线索未被充分利用，且这种不平衡随着灾难性遗忘的累积而加剧。为解决此问题，我们提出MAND，一种用于多模态自我中心开放世界持续学习的模态感知框架。在推理时，模态感知自适应评分（MoAS）利用样本级可靠性自适应调整模态贡献，并通过偏差和分歧惩罚细化新颖性评分。在训练时，模态感知表示稳定训练（MoRST）通过模态特定头和模态级logits蒸馏保留每个模态在任务间的判别能力。在公开多模态自我中心基准上的实验表明，MAND一致地提升了新颖活动检测和已知类准确率，同时大幅降低FPR95，表明更可靠的开放世界识别。源代码见\href{this https URL}{this http URL}。
英文摘要：
Multimodal egocentric activity recognition integrates visual and inertial cues for robust first-person behavior understanding. However, deploying such systems in open-world environments requires detecting novel activities while continuously learning from non-stationary data streams. Existing methods rely on the main fused logits for novelty scoring, without fully exploiting the complementary evidence available from individual modalities. Because these logits are often dominated by RGB, cues from other modalities, particularly IMU, remain underutilized, and this imbalance worsens as catastrophic forgetting accumulates. To address this, we propose MAND, a modality-aware framework for multimodal egocentric open-world continual learning. At inference, Modality-aware Adaptive Scoring (MoAS) adaptively adjusts modality contributions using sample-wise reliability and refines novelty scoring with deviation and disagreement penalties. During training, Modality-aware Representation Stabilization Training (MoRST) preserves the discriminative capacity of each modality across tasks through modality-specific heads and modality-wise logit distillation. Experiments on a public multimodal egocentric benchmark show that MAND consistently improves novel activity detection and known-class accuracy while substantially reducing FPR95, indicating more reliable open-world recognition. The source code is available at \href{https://github.com/HyeJeongIm/MAND}{github.com/HyeJeongIm/MAND}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
