LED：无需人工数据生成的LLM增强开放词汇目标检测
LED: LLM Enhanced Open-Vocabulary Object Detection without Human Curated Data Generation
AI总结：
本文提出LED方法，通过零初始化交叉注意力适配器将LLM隐藏状态直接融合到开放词汇目标检测器中，无需人工合成数据，在OmniLabel上显著提升GroundingDINO性能且计算开销极小。
摘要：
在大规模视觉语言数据上训练的大型基础模型可以通过合成训练数据提升开放词汇目标检测（OVD），然而手工设计的流程往往引入偏差并过拟合于特定提示。我们通过直接将大语言模型（LLM）的隐藏状态融合到检测器中来规避这一问题——这一途径出人意料地尚未被充分探索。本文提出了一种系统方法，通过利用多模态大语言模型（MLLM）中LLM的解码器层来增强视觉定位能力。我们引入了一个零初始化交叉注意力适配器，以实现从LLM到目标检测器的高效知识融合，这一新方法称为LED（LLM增强的开放词汇目标检测）。我们发现中间LLM层已经编码了丰富的空间语义；仅适配早期层即可获得大部分增益。以Swin-T作为视觉编码器，Qwen2-0.5B + LED在OmniLabel上将GroundingDINO提升了3.82%，而额外GFLOPs仅为8.7%；更大的视觉骨干网络将改进幅度提升至6.22%。关于适配器变体、LLM规模和融合深度的广泛消融实验进一步验证了我们的设计。
英文摘要：
Large foundation models trained on large-scale vision-language data can boost Open-Vocabulary Object Detection (OVD) via synthetic training data, yet the hand-crafted pipelines often introduce bias and overfit to specific prompts. We sidestep this issue by directly fusing hidden states from Large Language Models (LLMs) into detectors-an avenue surprisingly under-explored. This paper presents a systematic method to enhance visual grounding by utilizing decoder layers of the LLM of an MLLM. We introduce a zero-initialized cross-attention adapter to enable efficient knowledge fusion from LLMs to object detectors, a new approach called LED (LLM Enhanced Open-Vocabulary Object Detection). We find that intermediate LLM layers already encode rich spatial semantics; adapting only the early layers yields most of the gain. With Swin-T as the vision encoder, Qwen2-0.5B + LED lifts GroundingDINO by 3.82 % on OmniLabel at just 8.7 % extra GFLOPs, and a larger vision backbone pushes the improvement to 6.22 %. Extensive ablations on adapter variants, LLM scales and fusion depths further corroborate our design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
