相同还是不同？提升视觉语言模型的视觉感知能力
Same or Not? Enhancing Visual Perception in Vision-Language Models
AI总结：
TWIN通过引入大规模图像对数据集提升视觉语言模型的细粒度视觉感知能力，显著提高在艺术、动物等未见领域识别性能。
摘要：
视觉语言模型（VLMs）在广泛的视觉理解方面表现出色，但仍然存在粒度粗、存在视觉偏差以及遗漏细微视觉细节的问题。现有的训练语料强化了这一限制，因为它强调一般识别（“是猫还是狗？”）而不是细粒度感知。为了解决这个问题，我们引入了一个新的训练语料和任务，旨在增强VLMs的感知能力。TWIN是一个包含561,000对图像查询的大规模数据集，要求模型判断两幅视觉相似的图像是否描绘同一个物体，鼓励关注细微的视觉线索。该数据集涵盖了各种日常物体在不同情境、视角和外观下的广泛范围。在TWIN上微调VLMs在细粒度识别方面取得了显著提升，即使在未见过的领域如艺术、动物、植物和地标上也是如此。为了量化这些提升，我们引入了FGVQA，一个包含12,000个查询的基准测试集，它重新利用了多个领域的细粒度识别和检索数据集。尽管现有的VLMs在FGVQA上表现不佳，但当在TWIN上微调时，它们的性能提高了高达19.3%，而不会影响在通用VQA基准测试上的表现。最后，我们的TWIN数据集在物体注释方面具有良好的可扩展性，我们的分析表明规模是性能的关键。我们设想TWIN作为开源VLM训练语料的补充，推动未来模型的感知精度。项目网页：https://glab-caltech.github.io/twin/
英文摘要：
Vision-language models (VLMs) excel at broad visual understanding but remain coarse-grained, exhibit visual biases, and miss subtle visual details. Existing training corpora reinforce this limitation by emphasizing general recognition ("Is it a cat or a dog?") over fine-grained perception. To address this, we introduce a new training corpus and task designed to enhance the perceptual abilities of VLMs. TWIN is a large-scale dataset of 561,000 image-pair queries that task models to determine whether two visually similar images depict the same object, encouraging attention to nuanced visual cues. The dataset spans a diverse range of everyday objects across contexts, viewpoints, and appearances. Fine-tuning VLMs on TWIN yields notable gains in fine-grained recognition, even on unseen domains such as art, animals, plants, and landmarks. To quantify these gains, we introduce FGVQA, a benchmark suite of 12,000 queries that repurposes fine-grained recognition and retrieval datasets from multiple domains. While existing VLMs struggle on FGVQA, when fine-tuned on TWIN they improve by up to 19.3%, without compromising performance on general VQA benchmarks. Finally, our TWIN dataset scales favorably with object annotations, and our analysis shows that scale is key to performance. We envision TWIN as a drop-in addition to open-source VLM training corpora, advancing perceptual precision of future models. Project webpage: https://glab-caltech.github.io/twin/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
