ColorBlindnessEval：视觉语言模型能否通过色盲测试？
ColorBlindnessEval: Can Vision-Language Models Pass Color Blindness Tests?
AI总结：
本文提出ColorBlindnessEval基准，通过500张类石原图像评估9个VLMs在视觉对抗场景中的鲁棒性，发现其数字识别能力有限且存在幻觉问题，为提升VLMs在复杂视觉环境中的可靠性提供工具与依据。
摘要：
本文提出ColorBlindnessEval，这是一个新颖的基准，旨在评估视觉语言模型（VLMs）在受石原色盲测试启发的视觉对抗场景中的鲁棒性。我们的数据集包含500张类石原图像，这些图像以不同颜色组合呈现0到99的数字，挑战VLMs准确识别嵌入在复杂视觉模式中的数字信息。我们使用是/否和开放式提示评估9个VLMs，并将其性能与人类参与者进行比较。实验揭示了模型在对抗性语境中解释数字的能力存在局限性，突出了普遍的幻觉问题。这些发现强调了提高VLMs在复杂视觉环境中鲁棒性的必要性。ColorBlindnessEval可作为基准测试的宝贵工具，助力提升VLMs在准确性至关重要的实际应用中的可靠性。
英文摘要：
This paper presents ColorBlindnessEval, a novel benchmark designed to evaluate the robustness of Vision-Language Models (VLMs) in visually adversarial scenarios inspired by the Ishihara color blindness test. Our dataset comprises 500 Ishihara-like images featuring numbers from 0 to 99 with varying color combinations, challenging VLMs to accurately recognize numerical information embedded in complex visual patterns. We assess 9 VLMs using Yes/No and open-ended prompts and compare their performance with human participants. Our experiments reveal limitations in the models' ability to interpret numbers in adversarial contexts, highlighting prevalent hallucination issues. These findings underscore the need to improve the robustness of VLMs in complex visual environments. ColorBlindnessEval serves as a valuable tool for benchmarking and improving the reliability of VLMs in real-world applications where accuracy is critical.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
