用于缓解大型视觉语言模型中物体幻觉的黑盒视觉提示工程
Black-Box Visual Prompt Engineering for Mitigating Object Hallucination in Large Vision Language Models
AI总结：
针对大型视觉语言模型的物体幻觉问题，提出黑盒视觉提示工程框架，通过训练路由器模型动态选择最优视觉提示，无需访问模型内部即可有效缓解幻觉。
摘要：
大型视觉语言模型（LVLMs）常出现物体幻觉，这削弱了其可靠性。令人惊讶的是，我们发现简单的基于物体的视觉提示——在图像上叠加视觉线索（如边界框、圆圈）——能显著缓解此类幻觉；然而，不同的视觉提示（VPs）效果各异。为此，我们提出黑盒视觉提示工程（BBVPE），这是一个无需访问模型内部即可识别最优VPs以增强LVLM响应的框架。该方法利用候选VPs池并训练路由器模型，为给定输入图像动态选择最有效的VP。这种黑盒方法与模型无关，适用于开源和专有LVLMs。在POPE和CHAIR等基准上的评估表明，BBVPE有效减少了物体幻觉。
英文摘要：
Large Vision Language Models (LVLMs) often suffer from object hallucination, which undermines their reliability. Surprisingly, we find that simple object-based visual prompting -- overlaying visual cues (e.g., bounding box, circle) on images -- can significantly mitigate such hallucination; however, different visual prompts (VPs) vary in effectiveness. To address this, we propose Black-Box Visual Prompt Engineering (BBVPE), a framework to identify optimal VPs that enhance LVLM responses without needing access to model internals. Our approach employs a pool of candidate VPs and trains a router model to dynamically select the most effective VP for a given input image. This black-box approach is model-agnostic, making it applicable to both open-source and proprietary LVLMs. Evaluations on benchmarks such as POPE and CHAIR demonstrate that BBVPE effectively reduces object hallucination.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
