arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.07072cs.CV
收藏

用于半监督视频动作检测的稳定均值教师

Stable Mean Teacher for Semi-supervised Video Action Detection

University of Central Florida(中佛罗里达大学)

Akash Kumar, Sirshapan Mitra, Yogesh Singh Rawat

AI总结:

本文提出Stable Mean Teacher框架,通过Error Recovery模块和Difference of Pixels约束提升伪标签质量与时间一致性,在多个视频动作检测基准上显著优于监督基线。

摘要:

在这项工作中,我们关注视频动作检测的半监督学习。视频动作检测除了分类之外还需要时空定位,而有限的标签数量使得模型容易产生不可靠的预测。我们提出了Stable Mean Teacher,这是一个简单的端到端基于教师的框架,得益于改进且时间一致的伪标签。它依赖于一个新颖的Error Recovery (EoR)模块,该模块从学生在有标签样本上的错误中学习,并将这些知识转移给教师,以改进无标签样本的伪标签。此外,现有的时空损失没有考虑时间连贯性,容易出现时间不一致性。为了解决这个问题,我们提出了Difference of Pixels (DoP),这是一个简单且新颖的专注于时间一致性的约束,从而实现连贯的时间检测。我们在四个不同的时空检测基准上评估了我们的方法:UCF101-24、JHMDB21、AVA和YouTube-VOS。我们的方法在UCF101-24上平均超出监督基线23.5%,在JHMDB21上超出16%,在AVA上超出3.3%。仅使用10%和20%的数据,它在UCF101-24和JHMDB21上分别与在100%标注上训练的监督基线相比提供了有竞争力的性能。我们进一步在AVA上评估了其扩展到大规模数据集的有效性,以及在YouTube-VOS上评估视频目标分割的有效性,展示了其对视频领域其他任务的泛化能力。代码和模型已公开。

英文摘要:

In this work, we focus on semi-supervised learning for video action detection. Video action detection requires spatiotemporal localization in addition to classification, and a limited amount of labels makes the model prone to unreliable predictions. We present Stable Mean Teacher, a simple end-to-end teacher-based framework that benefits from improved and temporally consistent pseudo labels. It relies on a novel Error Recovery (EoR) module, which learns from students' mistakes on labeled samples and transfers this knowledge to the teacher to improve pseudo labels for unlabeled samples. Moreover, existing spatiotemporal losses do not take temporal coherency into account and are prone to temporal inconsistencies. To address this, we present Difference of Pixels (DoP), a simple and novel constraint focused on temporal consistency, leading to coherent temporal detections. We evaluate our approach on four different spatiotemporal detection benchmarks: UCF101-24, JHMDB21, AVA, and YouTube-VOS. Our approach outperforms the supervised baselines for action detection by an average margin of 23.5% on UCF101-24, 16% on JHMDB21, and 3.3% on AVA. Using merely 10% and 20% of data, it provides competitive performance compared to the supervised baseline trained on 100% annotations on UCF101-24 and JHMDB21, respectively. We further evaluate its effectiveness on AVA for scaling to large-scale datasets and YouTube-VOS for video object segmentation, demonstrating its generalization capability to other tasks in the video domain. Code and models are publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑