GlyphPattern：一个面向视觉语言模型的抽象模式识别基准
GlyphPattern: An Abstract Pattern Recognition Benchmark for Vision-Language Models
AI总结：
本文提出了 GlyphPattern 基准，通过包含 954 个项目、结合 40 种书写系统视觉模式描述的数据集，评估视觉语言模型的抽象模式识别能力，实验表明最先进模型如 GPT-4o 准确率仅 55%，揭示了模型在视觉处理、语言理解和模式泛化上的多重挑战。
摘要：
基于强大大型语言模型基础构建的视觉语言模型在跨视觉与文本数据的推理方面取得了快速进展。尽管 VLMs 在其训练过的视觉任务上表现良好，但我们的结果突出了抽象模式识别方面的关键挑战。我们提出了 GlyphPattern，一个包含 954 个项目的数据集，将来自 40 种书写系统的 318 项人类编写的视觉模式描述与三种视觉呈现风格相配对。GlyphPattern 评估 VLMs 中的抽象模式识别，要求模型理解并评判视觉模式的自然语言描述。GlyphPattern 模式取自一项针对人类书写系统的大规模认知科学研究；因此，它们富含空间指代与组合性。我们的实验表明，GlyphPattern 对最先进的 VLMs 具有挑战性（GPT-4o 仅达到 55% 的准确率），少样本提示仅带来微小的提升。我们详细的错误分析揭示了多个层面的挑战，包括视觉处理、自然语言理解和模式泛化。
英文摘要：
Vision-Language Models (VLMs) building upon the foundation of powerful large language models have made rapid progress in reasoning across visual and textual data. While VLMs perform well on vision tasks that they are trained on, our results highlight key challenges in abstract pattern recognition. We present GlyphPattern, a 954 item dataset that pairs 318 human-written descriptions of visual patterns from 40 writing systems with three visual presentation styles. GlyphPattern evaluates abstract pattern recognition in VLMs, requiring models to understand and judge natural language descriptions of visual patterns. GlyphPattern patterns are drawn from a large-scale cognitive science investigation of human writing systems; as a result, they are rich in spatial reference and compositionality. Our experiments show that GlyphPattern is challenging for state-of-the-art VLMs (GPT-4o achieves only 55% accuracy), with marginal gains from few-shot prompting. Our detailed error analysis reveals challenges at multiple levels, including visual processing, natural language understanding, and pattern generalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
