纹理还是语义？视觉语言模型在字体识别中迷失
Texture or Semantics? Vision-Language Models Get Lost in Font Recognition
AI总结：
该研究通过构建字体识别基准（FRB）评估视觉语言模型，发现其字体识别能力有限，易受斯特鲁普效应影响，少样本学习和思维链提示收效甚微。
摘要：
现代视觉语言模型（VLMs）展现出卓越的视觉和语言能力，在图像识别和物体定位等各类任务中取得了令人瞩目的性能。然而，它们在细粒度任务上的有效性仍是一个悬而未决的问题。在日常场景中，遇到设计材料（如杂志、排版教程、研究论文或品牌内容）的个人，可能希望识别文本中使用的外观美观的字体。鉴于其多模态能力和免费可访问性，许多视觉语言模型常被视为字体识别的潜在工具。这引出了一个根本性问题：视觉语言模型是否真正具备识别字体的能力？为了探究这一问题，我们引入了字体识别基准（FRB），这是一个紧凑且结构良好的数据集，包含15种常用字体。FRB包含两个版本：（i）简单版本，其中10个句子以不同字体渲染；（ii）困难版本，其中每个文本样本由15种字体本身的名称组成，引入了斯特鲁普效应，挑战模型的感知能力。通过对各种视觉语言模型在字体识别任务上的广泛评估，我们得出以下关键发现：（i）当前的视觉语言模型表现出有限的字体识别能力，许多最先进的模型未能达到令人满意的性能，并且容易受到文本信息引入的斯特鲁普效应的影响。（ii）少样本学习和思维链（CoT）提示在提高不同视觉语言模型的字体识别准确性方面提供的益处微乎其微。（iii）注意力分析揭示了视觉语言模型在捕捉语义特征方面的固有局限性。
英文摘要：
Modern Vision-Language Models (VLMs) exhibit remarkable visual and linguistic capabilities, achieving impressive performance in various tasks such as image recognition and object localization. However, their effectiveness in fine-grained tasks remains an open question. In everyday scenarios, individuals encountering design materials, such as magazines, typography tutorials, research papers, or branding content, may wish to identify aesthetically pleasing fonts used in the text. Given their multimodal capabilities and free accessibility, many VLMs are often considered potential tools for font recognition. This raises a fundamental question: Do VLMs truly possess the capability to recognize fonts? To investigate this, we introduce the Font Recognition Benchmark (FRB), a compact and well-structured dataset comprising 15 commonly used fonts. FRB includes two versions: (i) an easy version, where 10 sentences are rendered in different fonts, and (ii) a hard version, where each text sample consists of the names of the 15 fonts themselves, introducing a stroop effect that challenges model perception. Through extensive evaluation of various VLMs on font recognition tasks, we arrive at the following key findings: (i) Current VLMs exhibit limited font recognition capabilities, with many state-of-the-art models failing to achieve satisfactory performance and being easily affected by the stroop effect introduced by textual information. (ii) Few-shot learning and Chain-of-Thought (CoT) prompting provide minimal benefits in improving font recognition accuracy across different VLMs. (iii) Attention analysis sheds light on the inherent limitations of VLMs in capturing semantic features.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
