使用智能体工作流将语言与视觉物体重新对齐
Re-Aligning Language to Visual Objects with an Agentic Workflow
AI总结：
提出由LLM控制的智能体工作流Real-LOD，通过规划、工具使用和反思循环自适应调整提示，减少VLM幻觉并重新对齐语言与视觉物体，仅用0.18M图像训练即超越现有方法约50%。
摘要：
基于语言的目标检测（LOD）旨在将视觉物体与语言表达对齐。大量配对数据被用于提升LOD模型的泛化能力。在训练过程中，近期研究利用视觉语言模型（VLMs）自动为视觉物体生成类人表达，促进训练数据规模扩大。在此过程中，我们观察到VLM幻觉带来了不准确的物体描述（如物体名称、颜色和形状），从而降低了VL对齐质量。为减少VLM幻觉，我们提出了一种由LLM控制的智能体工作流，通过自适应调整图像和文本提示将语言与视觉物体重新对齐。我们将此工作流命名为Real-LOD，包含规划、工具使用和反思步骤。给定带有检测物体和VLM原始语言表达的图像，Real-LOD自动推理其状态并基于我们的神经符号设计安排动作（即规划）。该动作将自适应调整图像和文本提示，并将其发送给VLMs进行物体重新描述（即工具使用）。然后，我们使用另一个LLM分析这些精炼的表达以提供反馈（即反思）。这些步骤以循环形式进行，逐步改善语言描述以重新对齐视觉物体。我们构建了一个包含少量0.18M图像且带有重新对齐语言表达的数据集，并训练了一个主流LOD模型，在标准基准上以约50%的优势超越现有LOD方法。我们的Real-LOD工作流具有自动VL精炼功能，揭示了在扩大数据规模的同时保持数据质量的潜力，进一步从数据对齐角度提升了LOD性能。
英文摘要：
Language-based object detection (LOD) aims to align visual objects with language expressions. A large amount of paired data is utilized to improve LOD model generalizations. During the training process, recent studies leverage vision-language models (VLMs) to automatically generate human-like expressions for visual objects, facilitating training data scaling up. In this process, we observe that VLM hallucinations bring inaccurate object descriptions (e.g., object name, color, and shape) to deteriorate VL alignment quality. To reduce VLM hallucinations, we propose an agentic workflow controlled by an LLM to re-align language to visual objects via adaptively adjusting image and text prompts. We name this workflow Real-LOD, which includes planning, tool use, and reflection steps. Given an image with detected objects and VLM raw language expressions, Real-LOD reasons its state automatically and arranges action based on our neural symbolic designs (i.e., planning). The action will adaptively adjust the image and text prompts and send them to VLMs for object re-description (i.e., tool use). Then, we use another LLM to analyze these refined expressions for feedback (i.e., reflection). These steps are conducted in a cyclic form to gradually improve language descriptions for re-aligning to visual objects. We construct a dataset that contains a tiny amount of 0.18M images with re-aligned language expression and train a prevalent LOD model to surpass existing LOD methods by around 50% on the standard benchmarks. Our Real-LOD workflow, with automatic VL refinement, reveals a potential to preserve data quality along with scaling up data quantity, which further improves LOD performance from a data-alignment perspective.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
