arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.01245cs.CVcs.LG
收藏

SeFAR:基于时序扰动与学习稳定化的半监督细粒度动作识别

SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal Perturbation and Learning Stabilization

Unmanned System Research Institute, Northwestern Polytechnical University(西北工业大学无人系统研究院);School of Automation, Northwestern Polytechnical University(西北工业大学自动化学院);School of Computer Science, Northwestern Polytechnical University(西北工业大学计算机学院);School of Software, Northwestern Polytechnical University(西北工业大学软件学院)

Yongle Huang, Haodong Chen, Zhenbang Xu, Zihan Jia, Haozhou Sun, Dian Shao

AI总结:

本文提出半监督细粒度动作识别框架SeFAR,通过双层时序表示、时序扰动增强和自适应调节稳定学习,在多个细粒度与粗粒度数据集上取得最优性能。

摘要:

人类动作理解对于多模态系统的发展至关重要。尽管近期由强大大型语言模型(LLMs)驱动的发展旨在具备足够广泛的通用性以覆盖大量类别,但它们往往忽视了对更具体能力的需求。本文针对更具挑战性的细粒度动作识别(FAR)任务展开研究,该任务关注较短时间跨度内的细粒度语义标签(例如“向后团身转体1周的空翻”)。鉴于细粒度标签标注成本高昂,且微调LLMs需要大量数据,我们提出采用半监督学习(SSL)。我们的框架SeFAR融合了多项创新设计以应对这些挑战。具体而言,为捕获充分的视觉细节,我们构建了双层时序元素作为更有效的表示,并在此基础上设计了一种新的强增强策略,通过引入适度时序扰动用于教师-学生学习范式。此外,为应对教师模型在FAR预测中的高度不确定性,我们提出了自适应调节机制以稳定学习过程。实验表明,SeFAR在两个FAR数据集FineGym和FineDiving上,在不同数据规模下均取得了最先进的性能。在两个经典粗粒度数据集UCF101和HMDB51上,它也优于其他半监督方法。进一步的分析和消融研究验证了我们设计的有效性。此外,我们表明SeFAR提取的特征能够大幅提升多模态基础模型理解细粒度和领域特定语义的能力。

英文摘要:

Human action understanding is crucial for the advancement of multimodal systems. While recent developments, driven by powerful large language models (LLMs), aim to be general enough to cover a wide range of categories, they often overlook the need for more specific capabilities. In this work, we address the more challenging task of Fine-grained Action Recognition (FAR), which focuses on detailed semantic labels within shorter temporal duration (e.g., "salto backward tucked with 1 turn"). Given the high costs of annotating fine-grained labels and the substantial data needed for fine-tuning LLMs, we propose to adopt semi-supervised learning (SSL). Our framework, SeFAR, incorporates several innovative designs to tackle these challenges. Specifically, to capture sufficient visual details, we construct Dual-level temporal elements as more effective representations, based on which we design a new strong augmentation strategy for the Teacher-Student learning paradigm through involving moderate temporal perturbation. Furthermore, to handle the high uncertainty within the teacher model's predictions for FAR, we propose the Adaptive Regulation to stabilize the learning process. Experiments show that SeFAR achieves state-of-the-art performance on two FAR datasets, FineGym and FineDiving, across various data scopes. It also outperforms other semi-supervised methods on two classical coarse-grained datasets, UCF101 and HMDB51. Further analysis and ablation studies validate the effectiveness of our designs. Additionally, we show that the features extracted by our SeFAR could largely promote the ability of multimodal foundation models to understand fine-grained and domain-specific semantics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑