arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.07347cs.CV
收藏

基于多级同构架构的运动放大频率解耦

Frequency Decoupling for Motion Magnification via Multi-Level Isomorphic Architecture

School of Computer Science and Information Engineering, Hefei University of Technology(合肥工业大学计算机与信息工程学院);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院);School of Computer Science, University of Nottingham(诺丁汉大学计算机学院)

Fei Wang, Dan Guo, Kun Li, Zhun Zhong, Meng Wang

AI总结:

本文提出FD4MM,一种基于多级同构架构的运动放大频率解耦新范式,通过稀疏高/低通滤波器和稀疏频率混合器捕获多级高频细节与稳定低频运动场,并创新引入对比正则化减少不期望放大,在降低计算量和提升推理速度的同时超越了SOTA方法。

摘要:

视频运动放大(VMM)旨在揭示宏观世界中物体细微且难以察觉的运动信息。先前的方法通过分离形状和纹理的表示学习,或基于相位波动的多域学习,直接从欧拉视角对运动场进行建模。受频谱启发,我们观察到具有稳定能量的低频分量始终包含空间结构且噪声较少,使其适合对细微运动场进行建模。为此,我们提出了FD4MM,这是一种基于多级同构架构的运动放大频率解耦新范式,用于在视频空间中捕获多级高频细节和稳定的低频结构(运动场)。由于高频细节和细微运动因其固有的微妙性以及来自噪声的不可避免外部干扰,容易受到信息退化的影响,我们精心设计了稀疏高/低通滤波器以增强细节和运动结构的完整性,并设计了稀疏频率混合器以促进无缝重耦合。此外,我们为该任务创新性地设计了一种对比正则化,以增强模型区分无关特征的能力,减少不期望的运动放大。在真实世界和合成数据集上的大量实验表明,我们的FD4MM优于SOTA方法。同时,与最新方法相比,FD4MM减少了1.63倍的FLOPs,并将推理速度提升了1.68倍。我们的代码可在https://github.com/Jiafei127/FD4MM获取。

英文摘要:

Video Motion Magnification (VMM) aims to reveal subtle and imperceptible motion information of objects in the macroscopic world. Prior methods directly model the motion field from the Eulerian perspective by Representation Learning that separates shape and texture or Multi-domain Learning from phase fluctuations. Inspired by the frequency spectrum, we observe that the low-frequency components with stable energy always possess spatial structure and less noise, making them suitable for modeling the subtle motion field. To this end, we present FD4MM, a new paradigm of Frequency Decoupling for Motion Magnification with a Multi-level Isomorphic Architecture to capture multi-level high-frequency details and a stable low-frequency structure (motion field) in video space. Since high-frequency details and subtle motions are susceptible to information degradation due to their inherent subtlety and unavoidable external interference from noise, we carefully design Sparse High/Low-pass Filters to enhance the integrity of details and motion structures, and a Sparse Frequency Mixer to promote seamless recoupling. Besides, we innovatively design a contrastive regularization for this task to strengthen the model's ability to discriminate irrelevant features, reducing undesired motion magnification. Extensive experiments on both Real-world and Synthetic Datasets show that our FD4MM outperforms SOTA methods. Meanwhile, FD4MM reduces FLOPs by 1.63$\times$ and boosts inference speed by 1.68$\times$ than the latest method. Our code is available at https://github.com/Jiafei127/FD4MM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑