VRPTEST：评估大型多模态模型中的视觉指称提示
VRPTEST: Evaluating Visual Referring Prompting in Large Multimodal Models
AI总结：
本文提出 VRPTEST 基准与基于蜕变测试的自动评估框架，系统评测多种视觉指称提示策略对八类 LMM 的影响，发现专有模型平均领先 22.70%，且提示选择可造成 -17.5% 至 +7.3% 的准确率变化。
摘要：
随着大型多模态模型（LMM）近期在多个领域取得进展，一种名为视觉指称提示的新型提示方法已经出现，并在增强多模态系统中的人机交互方面展现出显著潜力。与传统文本描述或坐标相比，该方法为人类与这些系统交互提供了更自然、更灵活的途径。然而，视觉指称提示的分类仍未明确，其对 LMM 性能的影响也尚未得到正式检验。在本研究中，我们首次使用多种视觉指称提示策略对 LMM 进行了全面分析。我们引入了一个名为 VRPTEST 的基准数据集，包含 3 种不同的视觉任务和 2,275 张图像，涵盖提示策略的多种组合。利用 VRPTEST，我们对八个版本的主流开源和专有基础模型进行了综合评估，其中包括两个早期版本的 GPT-4V。我们开发了一个基于软件蜕变测试技术的自动化评估框架，无需人工干预或手动标注即可评估 LMM 的准确性。我们发现，当前专有模型总体上优于开源模型，平均准确率提升为 22.70%；但其仍有改进空间。此外，我们的定量分析表明，提示策略的选择会显著影响 LMM 的准确率，变化范围为 -17.5% 至 +7.3%。进一步的案例研究表明，合适的视觉指称提示策略能够提升 LMM 对上下文和位置信息的理解，而不合适的策略则可能导致回答被拒绝。我们还就如何尽量降低视觉指称提示对 LMM 的负面影响提供了见解。
英文摘要：
With recent advancements in Large Multimodal Models (LMMs) across various domains, a novel prompting method called visual referring prompting has emerged, showing significant potential in enhancing human-computer interaction within multimodal systems. This method offers a more natural and flexible approach to human interaction with these systems compared to traditional text descriptions or coordinates. However, the categorization of visual referring prompting remains undefined, and its impact on the performance of LMMs has yet to be formally examined. In this study, we conduct the first comprehensive analysis of LMMs using a variety of visual referring prompting strategies. We introduce a benchmark dataset called VRPTEST, comprising 3 different visual tasks and 2,275 images, spanning diverse combinations of prompt strategies. Using VRPTEST, we conduct a comprehensive evaluation of eight versions of prominent open-source and proprietary foundation models, including two early versions of GPT-4V. We develop an automated assessment framework based on software metamorphic testing techniques to evaluate the accuracy of LMMs without the need for human intervention or manual labeling. We find that the current proprietary models generally outperform the open-source ones, showing an average accuracy improvement of 22.70%; however, there is still potential for improvement. Moreover, our quantitative analysis shows that the choice of prompt strategy significantly affects the accuracy of LMMs, with variations ranging from -17.5% to +7.3%. Further case studies indicate that an appropriate visual referring prompting strategy can improve LMMs' understanding of context and location information, while an unsuitable one might lead to answer rejection. We also provide insights on minimizing the negative impact of visual referring prompting on LMMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
