基于统计矩和子空间描述子的自监督动作识别
Self-supervising Action Recognition by Statistical Moment and Subspace Descriptors
摘要：
在本文中，我们通过将RGB帧作为输入来学习预测动作概念和辅助描述符，对象描述符。所谓的幻觉流被训练来预测辅助线索，同时被馈送到分类层，然后在测试阶段被幻觉化以帮助网络。我们设计和幻觉两个描述符，一个利用四个流行的对象检测器应用于训练视频，另一个利用图像和视频级别的显着性检测器。第一个描述符对检测器和ImageNet类预测分数、置信度分数以及边界框和帧索引的空间位置进行编码，以捕获每个视频的特征的时空分布。另一个描述符编码显着图和强度图案的空间-角度梯度分布。受概率分布特征函数的启发，我们在上述中间描述符上捕获了四个统计矩。随着平均值、协方差、协扭性和协方差系数的数目线性地、二次地、三次地和四次地增长。特征向量的维数，我们通过协方差矩阵的前导n ′特征向量（所谓的子空间）来描述协方差矩阵，并且我们捕获偏度/峰度而不是代价高昂的余弦偏度/协峰度。我们在五个流行的数据集上获得了最先进的技术，如Charades和EPIC-Kitches。
英文摘要：
In this paper, we build on a concept of self-supervision by taking RGB frames as input to learn to predict both action concepts and auxiliary descriptors e.g., object descriptors. So-called hallucination streams are trained to predict auxiliary cues, simultaneously fed into classification layers, and then hallucinated at the testing stage to aid network. We design and hallucinate two descriptors, one leveraging four popular object detectors applied to training videos, and the other leveraging image- and video-level saliency detectors. The first descriptor encodes the detector- and ImageNet-wise class prediction scores, confidence scores, and spatial locations of bounding boxes and frame indexes to capture the spatio-temporal distribution of features per video. Another descriptor encodes spatio-angular gradient distributions of saliency maps and intensity patterns. Inspired by the characteristic function of the probability distribution, we capture four statistical moments on the above intermediate descriptors. As numbers of coefficients in the mean, covariance, coskewness and cokurtotsis grow linearly, quadratically, cubically and quartically w.r.t. the dimension of feature vectors, we describe the covariance matrix by its leading n' eigenvectors (so-called subspace) and we capture skewness/kurtosis rather than costly coskewness/cokurtosis. We obtain state of the art on five popular datasets such as Charades and EPIC-Kitchens.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
