非洲燕子还是欧洲燕子？面向细粒度目标分类的大视觉语言模型基准测试
African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification
AI总结：
本研究构建了细粒度目标分类多项选择基准FOCI，测试发现CLIP模型表现远超大视觉语言模型，指出其编码器与LLM细粒度对齐不足，需更多细粒度标注训练数据。
摘要：
近期的大视觉语言模型（LVLMs）在众多图像理解与推理任务上展现出出色能力。然而，细粒度目标分类任务（例如区分动物物种）尽管在下游应用中十分重要，相关研究仍不够充分。我们通过构建FOCI（细粒度目标分类，Fine-grained Object ClassIfication）填补了这一评估空白，这是一个基于现有目标分类数据集打造的高难度多项选择基准：（1）多项选择形式避免了将分类任务转化为开放式问答任务时产生的答案模糊问题；（2）我们通过CLIP模型挖掘负样本标签，保留了分类难度。FOCI在五个流行分类数据集的基础上，补充了来自ImageNet-21k的四个领域专属子集。我们在FOCI上对12个公开LVLM进行了基准测试，结果表明它测试的是与现有图像理解及推理基准互补的技能。关键的是，CLIP模型的表现远优于LVLM。由于LVLM的图像编码器源自这些CLIP模型，这说明编码器与大语言模型（LLM）之间在细粒度目标区分上的对齐不足，且需要具备更多细粒度标注的（预）训练数据。我们已在https://github.com/gregor-ge/FOCI-Benchmark 发布代码。
英文摘要：
Recent Large Vision-Language Models (LVLMs) demonstrate impressive abilities on numerous image understanding and reasoning tasks. The task of fine-grained object classification (e.g., distinction between \textit{animal species}), however, has been probed insufficiently, despite its downstream importance. We fill this evaluation gap by creating \texttt{FOCI} (\textbf{F}ine-grained \textbf{O}bject \textbf{C}lass\textbf{I}fication), a difficult multiple-choice benchmark for fine-grained object classification, from existing object classification datasets: (1) multiple-choice avoids ambiguous answers associated with casting classification as open-ended QA task; (2) we retain classification difficulty by mining negative labels with a CLIP model. \texttt{FOCI}\xspace complements five popular classification datasets with four domain-specific subsets from ImageNet-21k. We benchmark 12 public LVLMs on \texttt{FOCI} and show that it tests for a \textit{complementary skill} to established image understanding and reasoning benchmarks. Crucially, CLIP models exhibit dramatically better performance than LVLMs. Since the image encoders of LVLMs come from these CLIP models, this points to inadequate alignment for fine-grained object distinction between the encoder and the LLM and warrants (pre)training data with more fine-grained annotation. We release our code at \url{https://github.com/gregor-ge/FOCI-Benchmark}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
