PerceptionCLIP：通过推断并条件化上下文进行视觉分类
PerceptionCLIP: Visual Classification by Inferring and Conditioning on Contexts
AI总结：
提出无需训练的零样本分类方法PerceptionCLIP，先推断上下文属性再条件化分类，提升泛化性、群体鲁棒性和可解释性。
摘要：
像CLIP这样的视觉-语言模型因其理解各种视觉概念和自然语言描述的能力而被广泛用于零样本图像分类。然而，如何充分利用CLIP前所未有的类人理解能力以获得更好的性能仍是一个开放性问题。本文从人类视觉感知过程中汲取灵感：在对物体进行分类时，人类首先推断上下文属性（如背景和方向），这些属性有助于将前景物体与背景分离，然后基于这些信息对物体进行分类。受此启发，我们观察到为CLIP提供上下文属性可以改善零样本图像分类并减轻对虚假特征的依赖。我们还观察到CLIP本身能够从图像中合理推断出这些属性。基于这些观察，我们提出了一种无需训练、两步式的零样本分类方法PerceptionCLIP。给定一张图像，它首先推断上下文属性（如背景），然后基于这些属性进行物体分类。我们的实验表明，PerceptionCLIP实现了更好的泛化性、群体鲁棒性和可解释性。我们的代码可在https://github.com/umd-huang-lab/perceptionCLIP获取。
英文摘要：
Vision-language models like CLIP are widely used in zero-shot image classification due to their ability to understand various visual concepts and natural language descriptions. However, how to fully leverage CLIP's unprecedented human-like understanding capabilities to achieve better performance is still an open question. This paper draws inspiration from the human visual perception process: when classifying an object, humans first infer contextual attributes (e.g., background and orientation) which help separate the foreground object from the background, and then classify the object based on this information. Inspired by it, we observe that providing CLIP with contextual attributes improves zero-shot image classification and mitigates reliance on spurious features. We also observe that CLIP itself can reasonably infer the attributes from an image. With these observations, we propose a training-free, two-step zero-shot classification method PerceptionCLIP. Given an image, it first infers contextual attributes (e.g., background) and then performs object classification conditioning on them. Our experiments show that PerceptionCLIP achieves better generalization, group robustness, and interoperability. Our code is available at https://github.com/umd-huang-lab/perceptionCLIP
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
