SHiNe:面向开放词汇目标检测的语义层级联结体
SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detection
AI总结:
针对现有开放词汇目标检测器在不同语义粒度词汇下性能波动大的问题,提出无需训练的SHiNe分类器,通过融合层级语义句子嵌入生成分类向量,提升多粒度鲁棒性且可即插即用。
摘要:
开放词汇目标检测(OvOD)已将检测任务转化为语言引导型任务,支持用户在推理阶段自由定义感兴趣的类别词汇表。然而我们的初步研究表明,现有OvOD检测器在处理不同语义粒度的词汇表时表现出显著的性能波动,这给实际部署带来了隐患。为此,我们提出语义层级联结体(Semantic Hierarchy Nexus,SHiNe)——一种利用类别层级语义知识的新型分类器。它以离线三步流程运行:i)为每个目标类别从层级结构中检索相关的父类/子类;ii)将这些类别整合为感知层级信息的句子;iii)融合这些句子的嵌入向量生成联结体分类器向量。我们在多个检测基准上的评估显示,SHiNe提升了模型在不同词汇粒度下的鲁棒性:使用真实层级结构时,mAP50最高提升31.9%;使用大语言模型生成的层级结构时仍能保持性能提升。此外,将SHiNe应用于ImageNet-1k上的开放词汇分类任务时,它将CLIP零样本基线的准确率提升了2.8%。SHiNe无需训练,可无缝集成到任何现成的OvOD检测器中,且推理阶段不会产生额外计算开销。代码已开源。
英文摘要:
Open-vocabulary object detection (OvOD) has transformed detection into a language-guided task, empowering users to freely define their class vocabularies of interest during inference. However, our initial investigation indicates that existing OvOD detectors exhibit significant variability when dealing with vocabularies across various semantic granularities, posing a concern for real-world deployment. To this end, we introduce Semantic Hierarchy Nexus (SHiNe), a novel classifier that uses semantic knowledge from class hierarchies. It runs offline in three steps: i) it retrieves relevant super-/sub-categories from a hierarchy for each target class; ii) it integrates these categories into hierarchy-aware sentences; iii) it fuses these sentence embeddings to generate the nexus classifier vector. Our evaluation on various detection benchmarks demonstrates that SHiNe enhances robustness across diverse vocabulary granularities, achieving up to +31.9% mAP50 with ground truth hierarchies, while retaining improvements using hierarchies generated by large language models. Moreover, when applied to open-vocabulary classification on ImageNet-1k, SHiNe improves the CLIP zero-shot baseline by +2.8% accuracy. SHiNe is training-free and can be seamlessly integrated with any off-the-shelf OvOD detector, without incurring additional computational overhead during inference. The code is open source.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。