CollagePrompt：面向GPT-4V经济型视觉识别的基准
CollagePrompt: A Benchmark for Budget-Friendly Visual Recognition with GPT-4V
AI总结：
本文提出CollagePrompt基准，通过将多图拼贴为单一提示以降低GPT-4V视觉识别成本，并基于遗传算法优化拼贴布局以提升识别效率。
摘要：
生成式人工智能的最新进展表明，通过采用视觉提示，GPT-4V能够在视觉识别任务中展现出显著的能力。尽管其能力令人印象深刻，但GPT-4V推理所带来的财务成本对其广泛使用构成了重大障碍。为应对这一挑战，我们提出了一项经济型拼贴提示任务，该任务将多张图像拼贴到单个视觉提示中，并让GPT-4V同时对多张图像执行视觉识别，从而降低成本。我们收集了包含多种拼贴提示的数据集，以评估其在GPT-4V视觉识别中的表现。我们的评估揭示了几个关键发现：1）识别准确率随拼贴中不同位置而变化。2）将同一类别的图像分组在一起可获得更好的视觉识别结果。3）错误的标签通常来自相邻图像。这些发现凸显了拼贴提示中图像排列的重要性。为此，我们构建了一个名为CollagePrompt的基准，它为设计拼贴提示提供了一个平台，以实现利用GPT-4V进行更具成本效益的视觉识别。我们提出了一种源自遗传算法的基线方法来优化拼贴布局，并引入了两个指标来衡量优化后拼贴提示的效率。我们的基准使研究人员能够更好地优化拼贴提示，从而使GPT-4V在视觉识别中更具成本效益。代码和数据可在项目页面https://collageprompting.github.io/获取。
英文摘要：
Recent advancements in generative AI have suggested that by taking visual prompts, GPT-4V can demonstrate significant proficiency in visual recognition tasks. Despite its impressive capabilities, the financial cost associated with GPT-4V's inference presents a substantial barrier to its wide use. To address this challenge, we propose a budget-friendly collage prompting task that collages multiple images into a single visual prompt and makes GPT-4V perform visual recognition on several images simultaneously, thereby reducing the cost. We collect a dataset of various collage prompts to assess its performance in GPT-4V's visual recognition. Our evaluations reveal several key findings: 1) Recognition accuracy varies with different positions in the collage. 2) Grouping images of the same category together leads to better visual recognition results. 3) Incorrect labels often come from adjacent images. These findings highlight the importance of image arrangement within collage prompt. To this end, we construct a benchmark called CollagePrompt, which offers a platform for designing collage prompt to achieve more cost-effective visual recognition with GPT-4V. A baseline method derived from genetic algorithms to optimize collage layouts is proposed and two metrics are introduced to measure the efficiency of the optimized collage prompt. Our benchmark enables researchers to better optimize collage prompts, thus making GPT-4V more cost-effective in visual recognition. The code and data are available at this project page https://collageprompting.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
