arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.06201cs.CV
收藏

基于点监督的骨架人类动作分割

Point-Supervised Skeleton-Based Human Action Segmentation

Southeast University(东南大学)

Hongsong Wang, Yiqin Shen, Pengbo Yan, Jie Gui, Yang Zhang

AI总结:

本文提出了一种点监督框架,通过多模态骨架数据和新型原型相似性方法,实现了高效的基于骨架的人类动作分割,减少了注释工作量并提升了性能。

摘要:

基于骨架的时序动作分割是一项基础但具有挑战性的任务,在使智能系统感知和响应人类活动方面起着关键作用。虽然全监督方法取得了令人满意的性能,但它们需要昂贵的帧级注释,并且对模糊的动作边界敏感。为了解决这些问题,我们介绍了一种点监督框架用于基于骨架的动作分割,其中每个动作片段只标注一个帧。我们利用多模态骨架数据,包括关节、骨和运动信息,通过预训练的统一模型进行编码,以提取丰富的特征表示。为了生成可靠的伪标签,我们提出了一种新的原型相似性方法,并将其与两种现有方法:能量函数和约束K-均值聚类相结合。多模态伪标签整合被提出,以增强伪标签的可靠性并指导模型训练。我们建立了新的基准测试在PKU-MMD(X-Sub和X-View)、MCFS-22和MCFS-130上,并实现了点监督基于骨架的人类动作分割的基线。广泛的实验表明,我们的方法实现了具有竞争力的性能,即使在显著减少注释工作量的同时,也超越了一些全监督方法。

英文摘要:

Skeleton-based temporal action segmentation is a fundamental yet challenging task, playing a crucial role in enabling intelligent systems to perceive and respond to human activities. While fully-supervised methods achieve satisfactory performance, they require costly frame-level annotations and are sensitive to ambiguous action boundaries. To address these issues, we introduce a point-supervised framework for skeleton-based action segmentation, where only a single frame per action segment is labeled. We leverage multimodal skeleton data, including joint, bone, and motion information, encoded via a pretrained unified model to extract rich feature representations. To generate reliable pseudo-labels, we propose a novel prototype similarity method and integrate it with two existing methods: energy function and constrained K-Medoids clustering. Multimodal pseudo-label integration is proposed to enhance the reliability of the pseudo-label and guide the model training. We establish new benchmarks on PKU-MMD (X-Sub and X-View), MCFS-22, and MCFS-130, and implement baselines for point-supervised skeleton-based human action segmentation. Extensive experiments show that our method achieves competitive performance, even surpassing some fully-supervised methods while significantly reducing annotation effort.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑