MegaCOIN：增强视觉-语言模型的中粒度颜色感知能力
MegaCOIN: Enhancing Medium-Grained Color Perception for Vision-Language Models
AI总结：
本文提出含22万真实图像、66万条人工标注的MegaCOIN数据集与基准，用于评估并微调VLM的中粒度颜色和环境感知，且小模型微调后在部分场景超过GPT-4o。
摘要：
在视觉-语言模型（VLM）中，感知并解释颜色与物理环境的能力，对于实现符合情境的准确理解和交互至关重要。然而，尽管多模态建模已取得进展，目前仍严重缺乏专门数据集，用以严格评估模型辨别细微颜色变化和空间语境的能力；这些要素对于情境理解以及在真实应用中的可靠部署都十分关键。为实现这一目标，我们构建了MegaCOIN，这是一个基于真实图像并具有多种语境属性的高质量人工标注数据集。MegaCOIN由两部分组成：MegaCOIN-Instruct，用作VLM的监督微调（SFT）数据集；以及MegaCOIN-Bench，这是一个带标注的测试集，也可作为独立的问答数据集使用。MegaCOIN为220,000张真实图像提供三项标注特征：前景颜色、背景颜色以及物体物理环境的描述，共构成66万条人工标注。此外，MegaCOIN还可用于对领域泛化（DG）算法进行基准评测。我们探索了在VLM的线性探测设置下对DG方法进行基准测试，并展示了一些新的见解。最后同样重要的是，我们表明包括GPT-4o在内的VLM在颜色识别能力上表现欠佳，而使用MegaCOIN进行微调可以提升其在视觉评估任务上的性能。在某些情况下，经过MegaCOIN微调的小规模开源模型（如LLaVA和Bunny）能够优于闭源的GPT-4o。我们希望MegaCOIN的实用价值能够揭示VLM可改进的方向，并为领域泛化算法提供一个更复杂的平台。
英文摘要：
In vision-language models (VLMs), the ability to perceive and interpret color and physical environment is crucial for achieving contextually accurate understanding and interaction. However, despite advances in multimodal modeling, there remains a significant lack of specialized datasets that rigorously evaluate a model's capacity to discern subtle color variations and spatial context -- critical elements for situational comprehension and reliable deployment across real-world applications. Toward that goal, we curate MegaCOIN, a high-quality, human-labeled dataset based on \emph{real} images with various contextual attributes. MegaCOIN consists of two parts: MegaCOIN-Instruct, which serves as a supervised fine-tuning (SFT) dataset for VLMs; and MegaCOIN-Bench, an annotated test set that can be used as a stand-alone QA dataset. MegaCOIN~provides three annotated features for 220,000 real images: foreground color, background color, and description of an object's physical environment, constituting 660k human annotations. In addition, MegaCOIN can be applied to benchmark domain generalization (DG) algorithms. We explore benchmarking DG methods in the linear probing setup for VLM and show some new insights. Last but not least, we show that VLMs, including GPT-4o, have subpar color recognition capabilities, and fine-tuning with MegaCOIN can result in improved performance on visual evaluation tasks. In certain cases, MegaCOIN fine-tuned small-scale opensource models such as LLaVA and Bunny can outperform closed-source GPT-4o. We hope the utilities of MegaCOIN can shed light on the directions VLMs can improve and provide a more complex platform for domain generalization algorithms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
