UOUO：用于衡量视觉语言模型知识边界的去语境化罕见对象
UOUO: Uncontextualized Uncommon Objects for Measuring Knowledge Horizons of Vision Language Models
AI总结：
提出UOUO基准，系统测试视觉语言模型对罕见对象的处理能力，发现较小模型在长尾分布任务上显著表现不佳，并提供了可扩展的数据收集流水线。
摘要：
较小规模的视觉语言模型（VLMs）通常声称在通用领域的视觉定位和问答基准上与更大模型表现相当，同时在计算效率和存储方面具有优势。然而，它们处理属于数据分布长尾的罕见对象的能力却鲜为人知。为了严格评估这一方面，我们引入了“去语境化罕见对象”（UOUO）基准。该基准专注于系统性地测试大参数量和小参数量的VLMs在罕见和专业化对象上的表现。我们的综合分析表明，尽管较小的VLMs在常见数据集上保持竞争力，但在涉及罕见对象的任务中表现显著不佳。我们还提出了一种先进、可扩展的数据收集和清洗流水线，确保UOUO基准提供高质量、具挑战性的实例。这些发现强调了在评估VLMs的真实能力时考虑长尾分布的必要性。
英文摘要：
Smaller-scale Vision-Langauge Models (VLMs) often claim to perform on par with larger models in general-domain visual grounding and question-answering benchmarks while offering advantages in computational efficiency and storage. However, their ability to handle rare objects, which fall into the long tail of data distributions, is less understood. To rigorously evaluate this aspect, we introduce the "Uncontextualized Uncommon Objects" (UOUO) benchmark. This benchmark focuses on systematically testing VLMs with both large and small parameter counts on rare and specialized objects. Our comprehensive analysis reveals that while smaller VLMs maintain competitive performance on common datasets, they significantly underperform on tasks involving uncommon objects. We also propose an advanced, scalable pipeline for data collection and cleaning, ensuring the UOUO benchmark provides high-quality, challenging instances. These findings highlight the need to consider long-tail distributions when assessing the true capabilities of VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
