帮助 CLIP 既见森林又见树木：一种分解与描述方法
Helping CLIP See Both the Forest and the Trees: A Decomposition and Description Approach
AI总结：
针对 CLIP 偏重全局图像模式而忽视局部细节的问题，本文提出随机多裁剪增强方法以激活其局部特征分析能力，并在零样本、少样本和测试时自适应设置下验证了有效性。
摘要：
像 CLIP 这样的视觉-语言模型（VLM）通过对比学习实现跨模态语义对齐，展现出鲁棒的零样本泛化能力。然而，传统的提示工程主要依赖粗粒度的类别标签，忽视了细粒度的局部语义。现有方法假设 VLM 天然能够识别局部化的视觉细节，并试图通过用大语言模型生成的属性描述符来增强文本提示以改善分类。然而，我们的系统性实验揭示了关键局限：CLIP 对全局图像模式的强烈偏置阻碍了其处理局部化视觉描述符的能力。为解决这一根本性约束，我们提出了一种简单、有效且即插即用的方案，使 CLIP 能够“既见森林又见树木”。具体而言，我们采用随机多裁剪增强来激活 CLIP 在局部特征分析方面的潜在能力。通过仅裁剪部分区域，该方法有效约束了模型的感受野并重新校准其注意力机制，从而缓解其固有偏置。我们在零样本、少样本和测试时自适应设置下评估了所提方法，大量实验表明 D&D 取得了有前景的性能。
英文摘要：
Vision-Language Models (VLMs) like CLIP achieve cross-modal semantic alignment through contrastive learning, exhibiting robust zero-shot generalization. Traditional prompt engineering, however, predominantly relies on coarse-grained category labels, neglecting fine-grained local semantics. Existing approaches assume that VLMs inherently recognize localized visual details and attempt to enhance classification by augmenting text prompts with attribute descriptors generated by large language models. However, our systematic experiments reveal critical limitations: CLIP's strong bias toward global image patterns hinders its ability to process localized visual descriptors. To address this fundamental constraint, we propose a simple, effective, and plug-and-play solution that enables CLIP to ``See Both the Forest and the Trees." Specifically, we employ stochastic multi-crop augmentation to activate CLIP's latent capacity for localized feature analysis. By cropping only partial regions, the approach effectively constrains the model's receptive field and recalibrates its attention mechanism, thereby mitigating its inherent bias. We evaluate the proposed method under zero-shot, few-shot, and test-time adaptation settings, and extensive experiments demonstrate that D&D achieves promising performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
