FoRA：超越多模态孪生网络的低秩适应模型
FoRA: Low-Rank Adaptation Model beyond Multimodal Siamese Network
AI总结：
针对多模态目标检测中双流网络融合复杂且参数量大的问题，提出共享骨干与低秩模态适配器（LMA），并设计自适应秩分配策略，在DroneVehicle上实现10.4%精度提升并减少149M参数。
摘要：
多模态目标检测为在各种视觉条件下实现鲁棒检测提供了广阔前景。然而，现有的双流骨干网络面临着复杂融合和大量参数增加的挑战。这主要是由于多模态同质信息存在巨大的数据分布偏差。在本文中，我们提出了一种新颖的多模态目标检测器，名为具有共享骨干的低秩模态适配器（LMA）。共享参数增强了同质信息的一致性，而轻量级模态适配器则专注于模态独有的特征。此外，我们设计了一种自适应秩分配策略，以适应不同特征级别的不同异质性。在应用于两个多模态目标检测数据集时，实验验证了我们方法的有效性。值得注意的是，在DroneVehicle上，LMA实现了10.4%的精度提升，同时减少了149M参数。代码可在https://github.com/zyszxhy/FoRA获取。我们的工作于2024年4月提交至ACM MM，但被拒绝。接下来我们将继续完善工作和论文撰写，主要包括FoRA的理论证明和多任务应用。
英文摘要：
Multimodal object detection offers a promising prospect to facilitate robust detection in various visual conditions. However, existing two-stream backbone networks are challenged by complex fusion and substantial parameter increments. This is primarily due to large data distribution biases of multimodal homogeneous information. In this paper, we propose a novel multimodal object detector, named Low-rank Modal Adaptors (LMA) with a shared backbone. The shared parameters enhance the consistency of homogeneous information, while lightweight modal adaptors focus on modality unique features. Furthermore, we design an adaptive rank allocation strategy to adapt to the varying heterogeneity at different feature levels. When applied to two multimodal object detection datasets, experiments validate the effectiveness of our method. Notably, on DroneVehicle, LMA attains a 10.4% accuracy improvement over the state-of-the-art method with a 149M-parameters reduction. The code is available at https://github.com/zyszxhy/FoRA. Our work was submitted to ACM MM in April 2024, but was rejected. We will continue to refine our work and paper writing next, mainly including proof of theory and multi-task applications of FoRA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
