合成刺激，真实收益：通过完全受控的数据生成重新思考VLM微调
Synthetic Stimuli, Real Gains: Rethinking VLM Fine-Tuning Through Fully Controlled Data Generation
AI总结：
本文提出一种完全受控的数据生成与标注流程，用于微调视觉语言模型（VLM），通过平衡分布和干净标注消除偏差，在空间推理任务上仅用130个样本即可实现均匀性能，并在真实世界数据上提升13%的性能。
摘要：
通过微调获得的视觉语言模型（VLM）的性能提升通常基于对真实世界场景的临时数据收集和标注。尽管有所改进，但这一过程往往容易受到偏差、错误和分布不平衡的影响，导致过拟合和性能不平衡。虽然少数研究探索了合成数据生成，但它们通常缺乏对数据分布和标注质量的控制。在这项工作中，我们通过探索完全受控的数据生成和标注流程，重新评估了模型微调的潜力，获得了具有平衡分布和干净标注的无偏差数据。以识别物体绝对位置的空间推理任务作为用例，我们微调了最先进的VLM，并在合成和真实世界基准上进行了详尽的评估，包括对真实世界场景的可迁移性。我们的实验揭示了两个关键发现：1）在平衡数据上微调可以在视觉场景中产生均匀的性能，并且仅用130个样本就能缓解常见偏差；2）在合成刺激上微调使真实世界数据（COCO）的性能提升了13%，优于在完整COCO训练集上微调的模型。
英文摘要：
Performance gains of Vision Language Models (VLMs) obtained by fine-tuning are generally based on ad hoc data collection and annotation of real-world scenes. Despite the improvements, this process is often prone to biases, errors, and distribution imbalance, resulting in overfitting and imbalanced performance. Although a few studies have explored synthetic data generation, they typically lack control over data distribution and annotation quality. In this work, we re-evaluate the potential of model fine-tuning by exploring a fully controlled data generation and annotation pipeline, obtaining bias-free data with balanced distribution and clean annotations. Using the spatial reasoning task of identifying the absolute position of an object as a use case, we fine-tune state-of-the-art VLMs and conduct exhaustive evaluations on both synthetic and real-world benchmarks, including transferability to real-world scenes. Our experiments reveal two key findings: 1) fine-tuning on balanced data yields uniform performance across the visual scene and mitigates common biases with as few as 130 samples; and 2) fine-tuning on synthetic stimuli improves performance by 13% on real-world data (COCO), outperforming models fine-tuned on the full COCO train set.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
