用CNN实现IDT描述符和I3D光流特征用于动作识别
Hallucinating IDT Descriptors and I3D Optical Flow Features for Action Recognition with CNNs
摘要：
在本文中，我们重新使用老式的手工制作的视频表示进行动作识别，并通过基于CNN的幻觉步骤为这些技术注入新的活力。尽管使用RGB和光流帧，I3 D模型（除其他外）在将其输出与改进的密集轨迹（IDT）相结合并通过词袋（BoW）和Fisher矢量（FV）编码的低级视频描述符提取时蓬勃发展。由于预处理、描述符提取、编码和调整参数，CNN和手工表示的这种融合是耗时的。因此，我们提出了一个端到端的可训练网络，它具有在训练阶段学习基于IDT的BoW/FV表示的流，并且易于与I3 D模型集成。具体来说，每个流在最后一个1D转换之前采用I3 D特征图。层，并学习将这些映射“翻译”为BoW/FV表示。因此，我们的模型可以产生幻觉并在测试阶段使用这种合成的BoW/FV表示。我们表明，甚至整个I3 D光流流的特征也可以被幻觉化，从而简化了流水线。我们的模型节省了20- 55小时的计算，并在四个公开的数据集上产生了最先进的结果。
英文摘要：
In this paper, we revive the use of old-fashioned handcrafted video representations for action recognition and put new life into these techniques via a CNN-based hallucination step. Despite of the use of RGB and optical flow frames, the I3D model (amongst others) thrives on combining its output with the Improved Dense Trajectory (IDT) and extracted with its low-level video descriptors encoded via Bag-of-Words (BoW) and Fisher Vectors (FV). Such a fusion of CNNs and handcrafted representations is time-consuming due to pre-processing, descriptor extraction, encoding and tuning parameters. Thus, we propose an end-to-end trainable network with streams which learn the IDT-based BoW/FV representations at the training stage and are simple to integrate with the I3D model. Specifically, each stream takes I3D feature maps ahead of the last 1D conv. layer and learns to `translate' these maps to BoW/FV representations. Thus, our model can hallucinate and use such synthesized BoW/FV representations at the testing stage. We show that even features of the entire I3D optical flow stream can be hallucinated thus simplifying the pipeline. Our model saves 20-55h of computations and yields state-of-the-art results on four publicly available datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
