边界投票网络用于模糊感知的时间戳监督动作分割

中文摘要 AI 辅助

时间戳监督的动作分割旨在利用每个动作随机标注的一帧，对未修剪视频中的动作进行分割和分类。在此设定下，从时间戳标注中精确地定位动作边界至关重要，因为这能生成逐帧伪标签并应用已被充分研究的全监督训练。然而，现有方法在动作过渡区域中因判别性特征不足，难以处理边界定位中的固有不确定性。这种不精确的边界估计显著降低了在模糊动作过渡区域中生成的伪标签的稳定性和可靠性，从而导致训练出的分割模型性能下降。在本文中，我们引入了边界投票网络，通过将视频级别的全局先验知识分层传播到局部动作过渡区域，来缓解特征模糊性。通过在整个视频中生成关键动作表示作为投票，并针对动作过渡区域，所有投票协同贡献于动作过渡特征增强和边界定位细化。大量实验证明了我们的方法在GTEA、50Salads和Breakfast数据集上的有效性。

英文摘要

Timestamp-supervised action segmentation aims to segment and classify actions in untrimmed videos with a random frame annotated per action. Precisely localizing action boundaries from timestamp annotations is crucial for this setting, as it enables generating framewise pseudo-labels and applying the well-explored fully-supervised training. However, prevailing methods struggle with intrinsic uncertainty in boundary localization due to less discriminative features in action-transiting regions. This imprecise boundary estimation significantly reduces the stability and reliability of the generated pseudo-labels in ambiguous action-transiting regions, consequently resulting in performance deterioration of the trained segmentation models. In our paper, we introduce the boundary voting network that mitigates feature ambiguity by hierarchically propagating video-level global prior knowledge into local action-transiting regions. By generating key action representations as votes throughout the video and targeting action-transiting regions, all votes collaboratively contribute to action-transiting feature enhancement and boundary localization refinement. Extensive experiments demonstrate the effectiveness of our method on GTEA, 50Salads, and Breakfast datasets.

发表机构

• School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电气与电子工程学院）
• Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）
• Amazon（亚马逊公司）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

Accepted to TCSVT 2025