StreamFlow:面向视频序列的流线型多帧光流估计
StreamFlow: Streamlined Multi-Frame Optical Flow Estimation for Video Sequences
AI总结:
针对多帧光流估计中递归计算冗余问题,提出流线型批内多帧框架StreamFlow,结合ISC时空建模与GTR时序回归,在KITTI和Sintel上提升遮挡区域性能并将速度提高63.82%。
摘要:
连续帧之间的遮挡长期以来一直是光流估计中的重大挑战。遮挡引入的固有模糊性直接违反了亮度恒定约束,并严重阻碍了像素到像素的匹配。为了解决这一问题,多帧光流方法利用相邻帧来缓解局部模糊性。然而,以往的多帧方法主要采用递归光流估计,导致大量的计算重叠。相比之下,我们提出了一种流线型批内框架,消除了对大量冗余递归计算的需求,同时在批内估计约束下开发了有效的时空建模方法。具体而言,我们提出了一种面向视频输入的流线型批内多帧(SIM)流水线,达到了与两帧网络相似的时间效率水平。此外,我们引入了一种高效的积分时空一致性(ISC)建模方法,用于在编码阶段进行有效的时空建模,且不引入额外的参数开销。另外,我们设计了一个全局时序回归器(GTR),在解码阶段有效探索时序关系。得益于高效的SIM流水线和有效的模块,StreamFlow不仅在具有挑战性的KITTI和Sintel数据集上表现出色,尤其在遮挡区域有所改进,而且与以往的多帧方法相比,速度显著提升了63.82%。代码将很快在https://github.com/littlespray/StreamFlow上提供。
英文摘要:
Occlusions between consecutive frames have long posed a significant challenge in optical flow estimation. The inherent ambiguity introduced by occlusions directly violates the brightness constancy constraint and considerably hinders pixel-to-pixel matching. To address this issue, multi-frame optical flow methods leverage adjacent frames to mitigate the local ambiguity. Nevertheless, prior multi-frame methods predominantly adopt recursive flow estimation, resulting in a considerable computational overlap. In contrast, we propose a streamlined in-batch framework that eliminates the need for extensive redundant recursive computations while concurrently developing effective spatio-temporal modeling approaches under in-batch estimation constraints. Specifically, we present a Streamlined In-batch Multi-frame (SIM) pipeline tailored to video input, attaining a similar level of time efficiency to two-frame networks. Furthermore, we introduce an efficient Integrative Spatio-temporal Coherence (ISC) modeling method for effective spatio-temporal modeling during the encoding phase, which introduces no additional parameter overhead. Additionally, we devise a Global Temporal Regressor (GTR) that effectively explores temporal relations during decoding. Benefiting from the efficient SIM pipeline and effective modules, StreamFlow not only excels in terms of performance on the challenging KITTI and Sintel datasets, with particular improvement in occluded areas but also attains a remarkable $63.82\%$ enhancement in speed compared with previous multi-frame methods. The code will be available soon at https://github.com/littlespray/StreamFlow.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。