LLM-Optic：揭示大型语言模型在通用视觉定位中的能力
LLM-Optic: Unveiling the Capabilities of Large Language Models for Universal Visual Grounding
AI总结：
LLM-Optic利用大型语言模型作为文本和视觉定位器，增强现有视觉定位模型对复杂查询的理解，无需训练即可实现零样本通用视觉定位。
摘要：
视觉定位是一种将用户提供的文本查询与图像中特定区域相关联的重要工具。尽管视觉定位模型取得了进展，但其理解复杂查询的能力仍然有限。为了克服这一限制，我们引入了LLM-Optic，一种创新方法，利用大型语言模型（LLMs）作为光学透镜，增强现有视觉定位模型对复杂文本查询的理解能力，这些查询涉及复杂的文本结构、多个对象或对象空间关系，而当前模型在这些情况下表现不佳。LLM-Optic首先使用LLM作为文本定位器（Text Grounder）来解析复杂的文本查询，并准确识别用户意图定位的对象。然后，利用预训练的视觉定位模型，根据文本定位器精炼后的查询生成候选边界框。之后，LLM-Optic用数字标记对候选边界框进行标注，以建立文本与特定图像区域之间的联系，从而连接两种不同的模态。最后，它采用大型多模态模型（LMM）作为视觉定位器（Visual Grounder），选择与原始文本查询最匹配的标记候选对象。通过LLM-Optic，我们实现了通用视觉定位，能够检测任意人类语言输入所指定的任意对象。重要的是，我们的方法无需额外训练或微调即可实现这一增强。在多个具有挑战性的基准上的大量实验表明，LLM-Optic达到了最先进的零样本视觉定位能力。项目页面：https://haoyu-zhao.github.io/LLM-Optic.github.io/。
英文摘要：
Visual grounding is an essential tool that links user-provided text queries with query-specific regions within an image. Despite advancements in visual grounding models, their ability to comprehend complex queries remains limited. To overcome this limitation, we introduce LLM-Optic, an innovative method that utilizes Large Language Models (LLMs) as an optical lens to enhance existing visual grounding models in comprehending complex text queries involving intricate text structures, multiple objects, or object spatial relationships, situations that current models struggle with. LLM-Optic first employs an LLM as a Text Grounder to interpret complex text queries and accurately identify objects the user intends to locate. Then a pre-trained visual grounding model is used to generate candidate bounding boxes given the refined query by the Text Grounder. After that, LLM-Optic annotates the candidate bounding boxes with numerical marks to establish a connection between text and specific image regions, thereby linking two distinct modalities. Finally, it employs a Large Multimodal Model (LMM) as a Visual Grounder to select the marked candidate objects that best correspond to the original text query. Through LLM-Optic, we have achieved universal visual grounding, which allows for the detection of arbitrary objects specified by arbitrary human language input. Importantly, our method achieves this enhancement without requiring additional training or fine-tuning. Extensive experiments across various challenging benchmarks demonstrate that LLM-Optic achieves state-of-the-art zero-shot visual grounding capabilities. Project Page: https://haoyu-zhao.github.io/LLM-Optic.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
