视觉-语言模型的分类学感知评估
Taxonomy-Aware Evaluation of Vision-Language Models
AI总结：
本文提出基于分类学的分层精确率与召回率评估框架，用于衡量视觉-语言模型无约束文本预测的正确性与具体性，并分析现代VLM在细粒度视觉分类上的表现。
摘要：
当视觉-语言模型（VLM）被提示识别图像中描绘的实体时，它可能回答“我看到一棵针叶树”，而不是具体的标签“挪威云杉”。这对评估提出了两个问题：第一，无约束的生成文本需要映射到评估标签空间（即“针叶树”）。第二，有用的分类度量应当对不够具体但并非错误的答案给予部分得分（“挪威云杉”是“针叶树”的一种）。为满足这些要求，我们提出了一个框架，用于将无约束文本预测（例如由视觉-语言模型生成的预测）与分类学（taxonomy）进行对照评估。具体而言，我们提出使用分层精确率和召回率度量来评估预测相对于分类学的正确性水平和具体性水平。在实验上，我们首先表明现有的文本相似度度量不能很好地捕捉分类学相似性。然后，我们开发并比较了将文本型VLM预测映射到分类学上的不同方法。这使我们能够计算生成文本与真实标签之间的分层相似度度量。最后，我们基于所提出的分类学评估方案，在细粒度视觉分类任务上分析了现代VLM。
英文摘要：
When a vision-language model (VLM) is prompted to identify an entity depicted in an image, it may answer 'I see a conifer,' rather than the specific label 'norway spruce'. This raises two issues for evaluation: First, the unconstrained generated text needs to be mapped to the evaluation label space (i.e., 'conifer'). Second, a useful classification measure should give partial credit to less-specific, but not incorrect, answers ('norway spruce' being a type of 'conifer'). To meet these requirements, we propose a framework for evaluating unconstrained text predictions, such as those generated from a vision-language model, against a taxonomy. Specifically, we propose the use of hierarchical precision and recall measures to assess the level of correctness and specificity of predictions with regard to a taxonomy. Experimentally, we first show that existing text similarity measures do not capture taxonomic similarity well. We then develop and compare different methods to map textual VLM predictions onto a taxonomy. This allows us to compute hierarchical similarity measures between the generated text and the ground truth labels. Finally, we analyze modern VLMs on fine-grained visual classification tasks based on our proposed taxonomic evaluation scheme.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
