arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.11953cs.CV
收藏

基于帧-事件融合网络的高帧率任意点跟踪

Tracking Any Point with Frame-Event Fusion Network at High Frame Rate

College of Intelligence Science and Technology, National University of Defense Technology(国防科技大学智能科学学院)

Jiaxiong Liu, Bo Wang, Zhen Tan, Jinpu Zhang, Hui Shen, Dewen Hu

AI总结:

针对帧率限制导致高速场景下点跟踪不稳定问题,提出图像-事件融合跟踪器FE-TAP,结合EvoFusion模块与Transformer细化策略,实现高帧率鲁棒跟踪,在EDS数据集上期望特征年龄提升24%。

摘要:

基于图像帧的任意点跟踪受限于帧率,导致在高速场景下不稳定,并在实际应用中泛化能力有限。为克服这些限制,我们提出了一种图像-事件融合点跟踪器FE-TAP,它将图像帧的上下文信息与事件的高时间分辨率相结合,实现了在各种挑战性条件下的高帧率和鲁棒点跟踪。具体而言,我们设计了一个演化融合模块(EvoFusion)来建模由事件引导的图像生成过程。该模块能有效整合来自两种不同频率模态的有价值信息。为获得更平滑的点轨迹,我们采用了一种基于Transformer的细化策略,迭代更新点的轨迹和特征。大量实验表明,我们的方法优于最先进的方法,特别是在EDS数据集上将期望特征年龄提升了24%。最后,我们使用自行设计的高分辨率图像-事件同步设备,在真实驾驶场景中定性验证了算法的鲁棒性。我们的源代码将在https://github.com/ljx1002/FE-TAP发布。

英文摘要:

Tracking any point based on image frames is constrained by frame rates, leading to instability in high-speed scenarios and limited generalization in real-world applications. To overcome these limitations, we propose an image-event fusion point tracker, FE-TAP, which combines the contextual information from image frames with the high temporal resolution of events, achieving high frame rate and robust point tracking under various challenging conditions. Specifically, we designed an Evolution Fusion module (EvoFusion) to model the image generation process guided by events. This module can effectively integrate valuable information from both modalities operating at different frequencies. To achieve smoother point trajectories, we employed a transformer-based refinement strategy that updates the point's trajectories and features iteratively. Extensive experiments demonstrate that our method outperforms state-of-the-art approaches, particularly improving expected feature age by 24$\%$ on EDS datasets. Finally, we qualitatively validated the robustness of our algorithm in real driving scenarios using our custom-designed high-resolution image-event synchronization device. Our source code will be released at https://github.com/ljx1002/FE-TAP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑