arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.14973cs.CRcs.CV
收藏

视觉基础模型对常见扰动的鲁棒性

Robustness of Vision Foundation Models to Common Perturbations

Duke University(杜克大学);Ant Group(蚂蚁集团)

Hongbin Liu, Zhengyuan Jiang, Cheng Hong, Neil Zhenqiang Gong

AI总结:

研究探讨了视觉基础模型对常见扰动的鲁棒性,提出三种鲁棒性指标和五种数学性质,评估了六个行业级模型,发现其普遍不鲁棒,并提出改进方法。

摘要:

视觉基础模型输出图像的嵌入向量,可能受常见编辑操作(如JPEG压缩、亮度和对比度调整)影响。这些扰动会改变嵌入向量并影响下游任务性能。本文首次系统研究了基础模型对这些扰动的鲁棒性,提出了三种鲁棒性度量标准,并制定了五种期望的数学性质,分析了这些度量标准是否满足或违反这些性质。使用这些度量标准,我们评估了六个行业级基础模型(OpenAI、Meta)在九种常见扰动类别中的表现,发现它们普遍不鲁棒。我们还展示了常见扰动会降低下游应用性能(如分类准确率),并证明鲁棒性值可以预测性能影响。最后,我们提出了一种细调方法,在不牺牲实用性的情况下提高鲁棒性。

英文摘要:

A vision foundation model outputs an embedding vector for an image, which can be affected by common editing operations (e.g., JPEG compression, brightness, contrast adjustments). These common perturbations alter embedding vectors and may impact the performance of downstream tasks using these embeddings. In this work, we present the first systematic study on foundation models' robustness to such perturbations. We propose three robustness metrics and formulate five desired mathematical properties for these metrics, analyzing which properties they satisfy or violate. Using these metrics, we evaluate six industry-scale foundation models (OpenAI, Meta) across nine common perturbation categories, finding them generally non-robust. We also show that common perturbations degrade downstream application performance (e.g., classification accuracy) and that robustness values can predict performance impacts. Finally, we propose a fine-tuning approach to improve robustness without sacrificing utility.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑