arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2104.07135cs.CV
收藏

用于视频理解的自适应中间表示

Adaptive Intermediate Representations for Video Understanding

Google Research(谷歌研究院)

Juhana Kangaspunta, AJ Piergiovanni, Rico Jonschkowski, Michael Ryoo, Anelia Angelova

摘要:

视频理解的一种常见策略是通过融合从RGB帧和光流导出的特征来合并空间和运动信息。在这项工作中,我们引入了一种新的方式来利用语义分割作为视频理解的中间表示,并使用它的方式,不需要额外的标签。其次,我们提出了一个通用框架,该框架与最终的视频理解任务联合学习中间表示(光流和语义分割),并允许根据最终目标调整表示。尽管在网络中使用了中间表示,但在推理过程中,除了RGB序列之外,不需要任何额外的数据,从而能够使用单个网络进行有效的识别。最后,我们提出了一种方法来找到最佳的学习配置,通过搜索最佳的损失加权进化。我们获得了更强大的视觉表示的视频,导致性能增益超过国家的最先进的。

英文摘要:

A common strategy to video understanding is to incorporate spatial and motion information by fusing features derived from RGB frames and optical flow. In this work, we introduce a new way to leverage semantic segmentation as an intermediate representation for video understanding and use it in a way that requires no additional labeling. Second, we propose a general framework which learns the intermediate representations (optical flow and semantic segmentation) jointly with the final video understanding task and allows the adaptation of the representations to the end goal. Despite the use of intermediate representations within the network, during inference, no additional data beyond RGB sequences is needed, enabling efficient recognition with a single network. Finally, we present a way to find the optimal learning configuration by searching the best loss weighting via evolution. We obtain more powerful visual representations for videos which lead to performance gains over the state-of-the-art.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑