BACON：通过概念袋图提升图像描述的清晰度
BACON: Improving Clarity of Image Captions via Bag-of-Concept Graphs
AI总结：
针对VLM生成图像描述冗长难解析的问题，提出BACON提示方法，将描述分解为结构化元素并转为JSON，训练LLaVA生成器，提升清晰度，使GroundingDINO等模型无需训练即可在开放词汇检测中实现1.51倍召回率提升。
摘要：
大型视觉-语言模型的进步带来了精确、准确的图像描述，这对于推进多模态图像理解和处理至关重要。然而，这些描述往往包含冗长且交织的上下文，难以解析，并且经常忽略关键线索，这给像GroundingDINO和SDXL这样的模型带来了巨大障碍，因为它们缺乏充分利用密集描述所需的强大文本编码和句法分析能力。为了解决这个问题，我们提出了BACON，一种提示方法，将VLM生成的描述分解为解耦的、结构化的元素，如对象、关系、风格和主题。这种方法不仅减少了处理复杂上下文时的混淆，还允许高效地转换为JSON字典，使不具备语言处理能力的模型能够轻松访问关键信息。我们使用BACON和GPT-4V标注了100,000个图像-描述对，并在此数据集上训练了一个LLaVA描述器，使其能够在不依赖昂贵的GPT-4V的情况下生成BACON风格的描述。对整体质量、精确度和召回率以及用户研究的评估表明，所得到的描述模型在生成高质量描述方面始终优于其他SOTA VLM模型。此外，我们展示了BACON风格的描述在应用于各种模型时表现出更好的清晰度，使它们能够完成以前无法实现的任务或在无需训练的情况下超越现有的SOTA解决方案。例如，BACON风格的描述帮助GroundingDINO在开放词汇目标检测任务中比领先方法实现了1.51倍的更高召回率得分。
英文摘要：
Advancements in large Vision-Language Models have brought precise, accurate image captioning, vital for advancing multi-modal image understanding and processing. Yet these captions often carry lengthy, intertwined contexts that are difficult to parse and frequently overlook essential cues, posing a great barrier for models like GroundingDINO and SDXL, which lack the strong text encoding and syntax analysis needed to fully leverage dense captions. To address this, we propose BACON, a prompting method that breaks down VLM-generated captions into disentangled, structured elements such as objects, relationships, styles, and themes. This approach not only minimizes confusion from handling complex contexts but also allows for efficient transfer into a JSON dictionary, enabling models without linguistic processing capabilities to easily access key information. We annotated 100,000 image-caption pairs using BACON with GPT-4V and trained an LLaVA captioner on this dataset, enabling it to produce BACON-style captions without relying on costly GPT-4V. Evaluations of overall quality, precision, and recall-as well as user studies-demonstrate that the resulting caption model consistently outperforms other SOTA VLM models in generating high-quality captions. Besides, we show that BACON-style captions exhibit better clarity when applied to various models, enabling them to accomplish previously unattainable tasks or surpass existing SOTA solutions without training. For example, BACON-style captions help GroundingDINO achieve 1.51x higher recall scores on open-vocabulary object detection tasks compared to leading methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
