通过持续预测学习实现行动本地化
Action Localization through Continual Predictive Learning
摘要：
动作识别的问题涉及在视频中定位动作，在时间和空间上都在图像中。当前占主导地位的方法使用监督学习来解决这个问题，并且需要大量带注释的训练数据，其形式为感兴趣区域周围的帧级限界框注释。在本文中，我们提出了一种新的方法，持续学习的基础上，使用特征级的预测自我监督。它不需要在帧级边界框方面进行任何训练注释。该方法的灵感来自视觉事件感知的认知模型，提出了一种基于预测的事件理解方法。我们使用一堆LSTM与CNN编码器结合，以及新颖的注意力机制来对视频中的事件进行建模，并使用该模型来预测未来帧的高级特征。预测误差用于连续学习模型的参数。这种自监督框架并不像其他方法那样复杂，但在学习标记和定位的鲁棒视觉表示方面非常有效。应该注意的是，该方法以流式方式输出，仅需要通过视频的单次通过，使其适合于实时处理。我们在三个数据集上证明了这一点- UCF Sports，JHMDB和THUMOS'13，并表明所提出的方法优于弱监督和无监督基线，并与完全监督基线相比获得了有竞争力的性能。最后，我们表明，所提出的框架可以推广到自我中心的视频，并在无监督的注视预测中获得最先进的结果。
英文摘要：
The problem of action recognition involves locating the action in the video, both over time and spatially in the image. The dominant current approaches use supervised learning to solve this problem, and require large amounts of annotated training data, in the form of frame-level bounding box annotations around the region of interest. In this paper, we present a new approach based on continual learning that uses feature-level predictions for self-supervision. It does not require any training annotations in terms of frame-level bounding boxes. The approach is inspired by cognitive models of visual event perception that propose a prediction-based approach to event understanding. We use a stack of LSTMs coupled with CNN encoder, along with novel attention mechanisms, to model the events in the video and use this model to predict high-level features for the future frames. The prediction errors are used to continuously learn the parameters of the models. This self-supervised framework is not complicated as other approaches but is very effective in learning robust visual representations for both labeling and localization. It should be noted that the approach outputs in a streaming fashion, requiring only a single pass through the video, making it amenable for real-time processing. We demonstrate this on three datasets - UCF Sports, JHMDB, and THUMOS'13 and show that the proposed approach outperforms weakly-supervised and unsupervised baselines and obtains competitive performance compared to fully supervised baselines. Finally, we show that the proposed framework can generalize to egocentric videos and obtain state-of-the-art results in unsupervised gaze prediction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
