arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2211.08252cs.CV
收藏

视频模型中的动态时域滤波

Dynamic Temporal Filtering in Video Models

JD Explore Academy(京东探索研究院);Singapore Management University(新加坡管理大学)

Fuchen Long, Zhaofan Qiu, Yingwei Pan, Ting Yao, Chong-Wah Ngo, Tao Mei

摘要:

传统的视频时间动态模型是用3D时空核或由2D空间核和1D时间核组成的分解版本来建模的。然而,建模能力受到固定窗口大小和内核沿时间维度的静态权重的限制。预先确定的核大小严重限制了时间感受野,并且固定权重平等地对待跨帧的每个空间位置,导致自然场景中的长距离时间建模的次优解决方案。在本文中,我们提出了一个新的配方的时间特征学习,即动态时间滤波器(DTF),新颖地执行空间感知的时间建模在频域与大的时间感受野。具体来说,DTF动态地为每个空间位置学习一个专门的频率滤波器,以模拟其长期的时间动态。同时,每个空间位置的时间特征也通过1D快速傅立叶变换(FFT)转换为频率特征谱。频谱由学习的频率滤波器调制,然后用逆FFT变换回时域。此外,为了方便DTF中频率滤波器的学习,我们执行逐帧聚合,通过帧间相关来增强主要时间特征及其时间邻居。将DTF块插入ConvNets和Transformer中,得到DTF-Net和DTF-Transformer是可行的。在三个数据集上进行的大量实验证明了我们的建议的优越性。更值得注意的是,DTF-Transformer在Kinetics-400数据集上实现了83.5%的准确率。源代码可从\url{https://github.com/FuchenUSTC/DTF}获得。

英文摘要:

Video temporal dynamics is conventionally modeled with 3D spatial-temporal kernel or its factorized version comprised of 2D spatial kernel and 1D temporal kernel. The modeling power, nevertheless, is limited by the fixed window size and static weights of a kernel along the temporal dimension. The pre-determined kernel size severely limits the temporal receptive fields and the fixed weights treat each spatial location across frames equally, resulting in sub-optimal solution for long-range temporal modeling in natural scenes. In this paper, we present a new recipe of temporal feature learning, namely Dynamic Temporal Filter (DTF), that novelly performs spatial-aware temporal modeling in frequency domain with large temporal receptive field. Specifically, DTF dynamically learns a specialized frequency filter for every spatial location to model its long-range temporal dynamics. Meanwhile, the temporal feature of each spatial location is also transformed into frequency feature spectrum via 1D Fast Fourier Transform (FFT). The spectrum is modulated by the learnt frequency filter, and then transformed back to temporal domain with inverse FFT. In addition, to facilitate the learning of frequency filter in DTF, we perform frame-wise aggregation to enhance the primary temporal feature with its temporal neighbors by inter-frame correlation. It is feasible to plug DTF block into ConvNets and Transformer, yielding DTF-Net and DTF-Transformer. Extensive experiments conducted on three datasets demonstrate the superiority of our proposals. More remarkably, DTF-Transformer achieves an accuracy of 83.5% on Kinetics-400 dataset. Source code is available at \url{https://github.com/FuchenUSTC/DTF}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑