视觉数据类型理解不会从视觉语言模型的规模化中涌现
Visual Data-Type Understanding does not emerge from Scaling Vision-Language Models
AI总结：
本文提出视觉数据类型识别任务，构建含27种修改类型的动物图像数据集并评估39个VLM，发现模型规模化无法带来该能力，而在微调描述中加入数据类型信息可显著提升性能。
摘要：
视觉语言模型（VLM）开发的最新进展在识别视觉语义内容方面取得了显著成功，包括在组合式图像理解方面令人印象深刻的实例。本文提出了视觉数据类型识别这一新任务，它是一项基础感知技能，对数据治理（例如从大型数据集中去除噪声数据、特定领域检索）和自主视觉（例如区分不断变化的天气条件与相机镜头污渍）具有重要意义。我们构建了两个数据集，其中包含经过27种多样视觉数据类型修改的动物图像，这些类型涵盖四大类别。对参数规模从100M到80B的39个VLM进行的广泛零样本评估揭示了细致的性能图景。尽管VLM在识别某些风格化数据类型（如卡通和素描）方面表现尚可，但它们在处理由图像旋转或加性噪声等基本操作产生的更简单数据类型时却遇到困难。我们的发现表明：（i）单纯扩大模型规模只会为CLIP等对比训练模型带来边际收益；（ii）对于OpenFlamingo等规模最大的自回归训练VLM，性能会出现明显下降。这一发现指出了当前前沿VLM的一个盲点：它们擅长识别语义内容，却无法通过规模化获得对视觉数据类型的理解。通过分析这些模型的预训练分布，并在微调过程中将数据类型信息加入描述文本，我们实现了显著的性能提升。通过探索这一此前未知的任务，我们旨在为进一步推进VLM、使其具备视觉数据类型理解奠定基础。代码和数据集发布于https://github.com/bethgelab/DataTypeIdentification。
英文摘要：
Recent advances in the development of vision-language models (VLMs) are yielding remarkable success in recognizing visual semantic content, including impressive instances of compositional image understanding. Here, we introduce the novel task of Visual Data-Type Identification, a basic perceptual skill with implications for data curation (e.g., noisy data-removal from large datasets, domain-specific retrieval) and autonomous vision (e.g., distinguishing changing weather conditions from camera lens staining). We develop two datasets consisting of animal images altered across a diverse set of 27 visual data-types, spanning four broad categories. An extensive zero-shot evaluation of 39 VLMs, ranging from 100M to 80B parameters, shows a nuanced performance landscape. While VLMs are reasonably good at identifying certain stylistic \textit{data-types}, such as cartoons and sketches, they struggle with simpler data-types arising from basic manipulations like image rotations or additive noise. Our findings reveal that (i) model scaling alone yields marginal gains for contrastively-trained models like CLIP, and (ii) there is a pronounced drop in performance for the largest auto-regressively trained VLMs like OpenFlamingo. This finding points to a blind spot in current frontier VLMs: they excel in recognizing semantic content but fail to acquire an understanding of visual data-types through scaling. By analyzing the pre-training distributions of these models and incorporating data-type information into the captions during fine-tuning, we achieve a significant enhancement in performance. By exploring this previously uncharted task, we aim to set the stage for further advancing VLMs to equip them with visual data-type understanding. Code and datasets are released at https://github.com/bethgelab/DataTypeIdentification.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
