arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2107.10068cs.CV
收藏

从单一到多重:利用多层次预测空间进行视频预测

From Single to Multiple: Leveraging Multi-level Prediction Spaces for Video Forecasting

Shenzhen Research Institute of Big Data(深圳大数据研究院);The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳))

Mengcheng Lan, Shuliang Ning, Yanran Li, Qian Chen, Xunlai Chen, Xiaoguang Han, Shuguang Cui

摘要:

尽管近年来视频预测已经成为一个广泛探索的话题,但现有的主流工作仍然将其模型限制在单一预测空间,而完全忽略了如何利用其模型与多预测空间。这项工作填补了这一空白。我们首次深入研究了在多预测空间中执行视频预测的多种策略,并将其结果融合在一起以提高性能。像素空间的预测通常缺乏保持视频语义和结构内容的能力,而高层特征空间的预测在约简和恢复过程中容易产生错误。因此,我们在不同的特征空间之间建立了一个循环连接,并将它们的生成纳入上采样过程。令人惊讶的是,这个简单的想法产生了比PhyDNet更显着的性能提升(MNIST-2数据集上的性能提高了32.1%MAE,KTH数据集上的性能提高了21.4%MAE)。在四个数据集上的定性和定量评价表明了该方法的泛化能力和有效性。我们表明,我们的模型显着减少了麻烦的失真和模糊的文物,并带来了显着的改善,在长期的视频预测的准确性。代码将很快发布。

英文摘要:

Despite video forecasting has been a widely explored topic in recent years, the mainstream of the existing work still limits their models with a single prediction space but completely neglects the way to leverage their model with multi-prediction spaces. This work fills this gap. For the first time, we deeply study numerous strategies to perform video forecasting in multi-prediction spaces and fuse their results together to boost performance. The prediction in the pixel space usually lacks the ability to preserve the semantic and structure content of the video however the prediction in the high-level feature space is prone to generate errors in the reduction and recovering process. Therefore, we build a recurrent connection between different feature spaces and incorporate their generations in the upsampling process. Rather surprisingly, this simple idea yields a much more significant performance boost than PhyDNet (performance improved by 32.1% MAE on MNIST-2 dataset, and 21.4% MAE on KTH dataset). Both qualitative and quantitative evaluations on four datasets demonstrate the generalization ability and effectiveness of our approach. We show that our model significantly reduces the troublesome distortions and blurry artifacts and brings remarkable improvements to the accuracy in long term video prediction. The code will be released soon.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑