利用大语言模型与布局到图像合成提升小样本检测性能
Boosting Few-Shot Detection with Large Language Models and Layout-to-Image Synthesis
AI总结：
针对小样本检测数据增强难题，提出结合LLM与布局到图像合成的协同框架，通过LLM外推空间先验生成新边界框，并用布局感知CLIP分数排序样本，在COCO小样本基准上大幅提升YOLOX-S的检测精度。
摘要：
扩散模型的最新进展催生了大量利用其生成海量高质量数据能力的研究，以服务于各类下游任务。这类模型的一个子类被称为布局到图像合成（Layout-to-Image Synthesis, LIS），它学习根据空间布局（边界框、掩码、姿态等）生成图像，尽管其布局遵循能力有限，但已展现出生成逼真图像的可观潜力。此外，如何有效迁移这类模型以实现小样本检测数据的可扩展增强，仍是一个悬而未决的问题。为此，我们提出一种协同框架，结合大语言模型（Large Language Model, LLM）与LIS模型来提升小样本检测性能，效果超越现有最先进的生成式增强方法。我们利用LLM的推理能力，仅通过少量示例标注就能生成新的边界框，从而外推出标注空间的空间先验。此外，我们提出了新颖的布局感知CLIP分数用于样本排序，实现了生成布局与图像之间的紧密耦合。在COCO小样本基准测试上，该方法取得了显著提升：采用我们的方法后，YOLOX-S基线模型在COCO 5-shot、10-shot和30-shot设置下的mAP分别提升了140%以上、50%和35%。
英文摘要：
Recent advancements in diffusion models have enabled a wide range of works exploiting their ability to generate high-volume, high-quality data for use in various downstream tasks. One subclass of such models, dubbed Layout-to-Image Synthesis (LIS), learns to generate images conditioned on a spatial layout (bounding boxes, masks, poses, etc.) and has shown a promising ability to generate realistic images, albeit with limited layout-adherence. Moreover, the question of how to effectively transfer those models for scalable augmentation of few-shot detection data remains unanswered. Thus, we propose a collaborative framework employing a Large Language Model (LLM) and an LIS model for enhancing few-shot detection beyond state-of-the-art generative augmentation approaches. We leverage LLM's reasoning ability to extrapolate the spatial prior of the annotation space by generating new bounding boxes given only a few example annotations. Additionally, we introduce our novel layout-aware CLIP score for sample ranking, enabling tight coupling between generated layouts and images. Significant improvements on COCO few-shot benchmarks are observed. With our approach, a YOLOX-S baseline is boosted by more than 140%, 50%, 35% in mAP on the COCO 5-,10-, and 30-shot settings, respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
