ColorFoil：研究大型视觉与语言模型中的色盲问题
ColorFoil: Investigating Color Blindness in Large Vision and Language Models
AI总结：
本文提出 ColorFoil 颜色干扰基准，在零样本设置下评测七种主流视觉语言模型，发现 ViLT 与 BridgeTower 的颜色感知显著优于 CLIP 系列和 GroupViT。
摘要：
借助 Transformer 架构，大型视觉与语言（V&L）模型即使在零样本设置下也表现出良好的性能。然而，若干研究表明，这些模型在处理复杂语言和视觉属性时缺乏鲁棒性。在本工作中，我们引入了一个新颖的 V&L 基准——ColorFoil，通过构造与颜色相关的 foil（干扰项）来评估模型对红、白、绿等颜色的感知与检测能力。我们在零样本设置下评估了七种最先进的 V&L 模型，包括 CLIP、ViLT、GroupViT 和 BridgeTower 等，并展示了来自这些 V&L 模型的有趣发现。实验评估表明，与 CLIP 及其变体以及 GroupViT 相比，ViLT 和 BridgeTower 表现出强得多的颜色感知能力。此外，基于 CLIP 的模型和 GroupViT 难以区分那些对具有正常颜色感知能力的人类而言在视觉上截然不同的颜色。
英文摘要：
With the utilization of Transformer architecture, large Vision and Language (V&L) models have shown promising performance in even zero-shot settings. Several studies, however, indicate a lack of robustness of the models when dealing with complex linguistics and visual attributes. In this work, we introduce a novel V&L benchmark - ColorFoil, by creating color-related foils to assess the models' perception ability to detect colors like red, white, green, etc. We evaluate seven state-of-the-art V&L models including CLIP, ViLT, GroupViT, and BridgeTower, etc. in a zero-shot setting and present intriguing findings from the V&L models. The experimental evaluation indicates that ViLT and BridgeTower demonstrate much better color perception capabilities compared to CLIP and its variants and GroupViT. Moreover, CLIP-based models and GroupViT struggle to distinguish colors that are visually distinct to humans with normal color perception ability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
