arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.07417cs.CVcs.LG
收藏

关于校准视觉-语言模型关键因素的实证研究

An Empirical Study Into What Matters for Calibrating Vision-Language Models

The Australian National University(澳大利亚国立大学);Curtin University(科廷大学);University of Óbuda(欧布达大学)

Weijie Tu, Weijian Deng, Dylan Campbell, Stephen Gould, Tom Gedeon

AI总结:

本文实证研究不同架构、数据集与训练策略下 VLM 的不确定性校准,发现 temperature scaling 即使面对分布和标签集变化也能稳定提升校准,且只需少量样本。

摘要:

视觉-语言模型(Vision-Language Models, VLMs)已成为零样本识别的主导方法,擅长处理多样化场景和显著的分布变化。然而,将其部署到风险敏感领域需要更深入地理解其不确定性估计能力,而这仍是一个相对未被探索的领域。在本研究中,我们考察了 VLMs 在不同架构、数据集和训练策略下的校准特性。具体而言,我们分析了当 VLMs 在某一领域、标签集或层级上进行校准,而在另一个不同的领域、标签集或层级上进行测试时,其不确定性估计性能。研究结果表明,尽管 VLMs 并非天然具备良好的不确定性校准,temperature scaling(温度缩放)能够显著且稳定地改善校准效果,即使在分布迁移和标签集变化的情况下也是如此。此外,VLMs 可以通过非常少量的样本完成校准。通过详尽实验,我们强调了这些发现的潜在应用和重要性,旨在推动 VLMs 在关键真实世界场景中得到更可靠、更有效的使用。

英文摘要:

Vision-Language Models (VLMs) have emerged as the dominant approach for zero-shot recognition, adept at handling diverse scenarios and significant distribution changes. However, their deployment in risk-sensitive areas requires a deeper understanding of their uncertainty estimation capabilities, a relatively uncharted area. In this study, we explore the calibration properties of VLMs across different architectures, datasets, and training strategies. In particular, we analyze the uncertainty estimation performance of VLMs when calibrated in one domain, label set or hierarchy level, and tested in a different one. Our findings reveal that while VLMs are not inherently calibrated for uncertainty, temperature scaling significantly and consistently improves calibration, even across shifts in distribution and changes in label set. Moreover, VLMs can be calibrated with a very small set of examples. Through detailed experimentation, we highlight the potential applications and importance of our insights, aiming for more reliable and effective use of VLMs in critical, real-world scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑