用于HAR的动态操纵超图：超越成对关系：基于视觉的人类活动识别的动态操纵超图

中文摘要 AI 辅助

细粒度操纵识别需要对双手、物体、工具和支撑表面之间不断演变的关系进行建模。传统的基于图的方法使用成对边，这可能会将一个协调事件分解为不相连的二元关系。我们提出了一个动态操纵超图框架，将多实体配置表示为高阶关系单元。在每个时间步，使用外观、空间、运动和语义角色特征对相关实体进行编码。使用接近度、接触和运动耦合谓词实例化并排序超边候选。超图推理网络执行节点到超边和超边到节点的消息传递，然后对不断演变的交互结构进行时间注意力。该框架提供与类别无关的超边重要性分数，可识别模型强调的实体配置和时间间隔，而无需将它们视为因果解释。在注释辅助实体定位协议下，在EPIC-KITCHENS-100/VISOR和Assembly101上进行了定量评估。仅视频和基于实体的方法提供了上下文比较，而匹配的成对图和静态超图作为主要的受控基线，因为它们使用相同的实体输入和可比的关系设置。所提出的方法在EPIC-KITCHENS-100/VISOR上比匹配的成对图提高了6.9个百分点的HO-F1，在Assembly101上提高了9.5个百分点，分别超过静态超图4.4和5.8个百分点。对ARCTIC的定性分析进一步表明，高排名超边与接触丰富的操纵间隔之间存在对应关系。这些结果证明了时变高阶关系建模对细粒度操纵活动识别的价值。

英文摘要

Fine-grained manipulation recognition requires modeling evolving relations among hands, objects, tools, and supporting surfaces. Conventional graph-based methods use pairwise edges that can fragment a coordinated event into disconnected binary relations. We propose a dynamic manipulation hypergraph framework that represents multi-entity configurations as higher-order relational units. At each temporal step, relevant entities are encoded using appearance, spatial, motion, and semantic-role features. Hyperedge candidates are instantiated and ranked using proximity, contact, and motion-coupling predicates. A hypergraph reasoning network performs node-to-hyperedge and hyperedge-to-node message passing, followed by temporal attention over the evolving interaction structure. The framework provides class-agnostic hyperedge-importance scores that identify entity configurations and temporal intervals emphasized by the model without treating them as causal explanations. Quantitative evaluation is conducted on EPIC-KITCHENS-100/VISOR and Assembly101 under an annotation-assisted entity-localization protocol. Video-only and entity-based methods provide contextual comparisons, while a matched pairwise graph and a static hypergraph serve as the principal controlled baselines because they use identical entity inputs and comparable relational settings. The proposed method improves HO-F1 over the matched pairwise graph by 6.9 percentage points on EPIC-KITCHENS-100/VISOR and 9.5 points on Assembly101, and exceeds the static hypergraph by 4.4 and 5.8 points, respectively. Qualitative analysis on ARCTIC further shows correspondence between highly ranked hyperedges and contact-rich manipulation intervals. These results demonstrate the value of time-varying higher-order relational modeling for fine-grained manipulation activity recognition.

发表机构

• School of Mathematics, Statistics and Computer Science, College of Science, University of Tehran（德黑兰大学理学院数学、统计与计算机科学学院）

机构由 AI 辅助整理，请以论文原文为准。