频率自适应的低延迟事件与帧目标检测
Frequency-Adaptive Low-Latency Object Detection Using Events and Frames
AI总结:
针对事件与RGB帧融合检测中的频率不匹配问题,提出FAOD,通过对齐模块和时间平移策略,以事件为主、RGB为辅,实现低延迟高频检测,在PKU-DAVIS-SOD上以四分之一参数提升mAP 9.8点。
摘要:
融合事件相机与RGB图像进行目标检测,利用了事件相机在恶劣环境下的鲁棒性以及RGB相机提供的丰富语义信息。然而,两个关键的不匹配问题严重阻碍了基于高频融合的目标检测:低延迟事件流与高延迟RGB帧之间的不匹配;训练时时间上稀疏的标签与推理时连续流之间的不匹配。为解决这些挑战,我们提出了频率自适应低延迟目标检测器(FAOD)。FAOD通过对齐模块将低频RGB帧与高频事件流对齐,该模块强化跨模态风格与空间邻近性以解决事件-RGB不匹配问题。我们进一步提出了一种训练策略——时间平移(Time Shift),该策略强制模块将来自时间平移的事件-RGB对及其原始表示的预测对齐,即与事件对齐的标注保持一致。该策略使网络能够以高频事件数据为主要参考,同时将低频RGB图像视为补充信息,从而保留事件流在面向高频检测时的低延迟特性。此外,我们观察到,与仅使用事件数据相比,这些校正后的事件-RGB对在从低训练频率向更高推理频率泛化时表现出更好的性能。在PKU-DAVIS-SOD和DSEC-Detection数据集上的大量实验表明,我们的FAOD达到了最先进的性能。具体而言,在PKU-DAVIS-SOD数据集中,FAOD在完全配对的事件-RGB数据上,以仅四分之一于SODFormer的参数实现了mAP提升9.8个百分点的成绩,并且在80倍事件-RGB频率不匹配下仍保持稳健性能(mAP仅下降3个百分点)。
英文摘要:
Fusing Events and RGB images for object detection leverages the robustness of Event cameras in adverse environments and the rich semantic information provided by RGB cameras. However, two critical mismatches: low-latency Events \textit{vs.}~high-latency RGB frames; temporally sparse labels in training \textit{vs.}~continuous flow in inference, significantly hinder the high-frequency fusion-based object detection. To address these challenges, we propose the \textbf{F}requency-\textbf{A}daptive Low-Latency \textbf{O}bject \textbf{D}etector (FAOD). FAOD aligns low-frequency RGB frames with high-frequency Events through an Align Module, which reinforces cross-modal style and spatial proximity to address the Event-RGB Mismatch. We further propose a training strategy, Time Shift, which enforces the module to align the prediction from temporally shifted Event-RGB pairs and their original representation, that is, consistent with Event-aligned annotations. This strategy enables the network to use high-frequency Event data as the primary reference while treating low-frequency RGB images as supplementary information, retaining the low-latency nature of the Event stream toward high-frequency detection. Furthermore, we observe that these corrected Event-RGB pairs demonstrate better generalization from low training frequency to higher inference frequencies compared to using Event data alone. Extensive experiments on the PKU-DAVIS-SOD and DSEC-Detection datasets demonstrate that our FAOD achieves SOTA performance. Specifically, in the PKU-DAVIS-SOD Dataset, FAOD achieves 9.8 points improvement in terms of the mAP in fully paired Event-RGB data with only a quarter of the parameters compared to SODFormer, and even maintains robust performance (only a 3 points drop in mAP) under 80$\times$ Event-RGB frequency mismatch.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。