EGM-Det：面向无人机RGB-IR目标检测的熵引导多模态自适应融合

中文摘要 AI 辅助

RGB与红外（IR）图像的联合使用可提升无人机视角下的目标检测性能，但现有多数方法采用静态或固定权重融合多模态特征，忽略了空间上变化的模态可靠性。本文提出EGM-Det，一种面向RGB-IR目标检测的熵引导多模态自适应融合框架。EGM-Det采用双流架构以保留模态专属表示，并引入熵偏移门融合模块用于自适应多尺度融合。该模块从输入强度、局部熵和跨模态差异中推导浅层熵先验，并用其引导局部偏移对齐与空间-通道门控融合，从而选择性聚合可靠的RGB和红外线索，而非均匀组合异构特征。本文进一步引入跨模态蒸馏以正则化学习到的融合门并减少融合退化：每个学生分支从与主分支匹配的跨模态教师分支中提取互补知识，同时熵自适应监督强调不确定的模态决策。在DroneVehicle、LLVIP和VEDAI数据集上的实验表明，EGM-Det在所有三个基准测试中均实现了最优性能，尤其在VEDAI上较现有方法提升超过10个百分点。

英文摘要

Joint use of RGB and infrared (IR) imagery can improve UAV-view object detection, but most existing methods fuse multimodal features with static or fixed weights and therefore overlook spatially varying modality reliability. We propose EGM-Det, an entropy-guided multimodal adaptive fusion framework for RGB-IR object detection. EGM-Det employs a dual-stream architecture to preserve modality-specific representations and introduces an Entropy Offset Gate Fusion module for adaptive multi-scale fusion. The module derives shallow entropy priors from input intensity, local entropy, and cross-modal discrepancy, and uses them to guide local offset alignment and spatial-channel gated fusion. It therefore selectively aggregates reliable RGB and infrared cues instead of uniformly combining heterogeneous features. We further introduce cross-modal distillation to regularize the learned fusion gates and reduce fusion degradation. Each student branch extracts complementary knowledge from the cross-modality teacher branch matched to the main branch, while entropy-adaptive supervision emphasizes uncertain modality decisions. Experiments on DroneVehicle, LLVIP, and VEDAI demonstrate state-of-the-art performance across all three benchmarks; in particular, EGM-Det outperforms prior approaches by more than 10 percentage points on VEDAI.

发表机构

• School of Cybersecurity, Northwestern Polytechnical University（西北工业大学网络空间安全学院）
• School of Automation and Software Engineering, Shanxi University（山西大学自动化与软件学院）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

14 pages, 7 figures, 6 tables