检索几何塑造基于缓存的CLIP适配
Retrieval Geometry Shapes Cache-Based Clip Adaptation
浏览论文内容
中文总结 AI 辅助
本研究揭示检索空间对缓存适配效果至关重要，提出MARC系统，利用DINOv2-B检索与CLIP预测，在多个分布偏移上显著提升性能且成本更低。
中文摘要 AI 辅助
基于缓存的测试时适配通过在保持模型冻结的同时存储和检索目标流中的样本来改进CLIP预测。然而，现有方法在很大程度上将用于图像-图像检索的特征空间视为固定的，从而留下了适配在多大程度上依赖于检索空间本身的问题。我们通过固定内存并仅改变检索编码器来研究这一问题，发现相同的内存可以产生截然不同的增益：在十六个检索空间中，ImageNet-A缓存增益从CLIP和MAE的至多+0.44点变化到DINOv2-L的+19.7±0.4点，而无标签的检索空间选择在ImageNet-V2上保留了98%的oracle增益。这些结果表明，内存质量不仅取决于存储了哪些样本，还取决于如何检索它们。受此发现启发，我们提出了MARC（Memory Augmented Retrieval for CLIP），一个无需训练的系统，使用冻结的CLIP进行预测，使用DINOv2-B进行检索，并采用单一融合权重。单视图缓存修复了1074±21个基线错误，而64视图集成修复了878±4个错误，成本约为其七分之一。在四个ImageNet分布偏移上，MARC达到了67.91%的OOD平均值，在匹配DINOv2-B规模和八视图下，实现了64.17±0.31%对比基于图的缓存系统的62.75±0.15%，同时运行速度快2.6倍。总体而言，我们的结果确立了检索空间作为在遥感、科学成像和变化视觉环境中稳健的基于缓存适配的一阶设计选择。
英文摘要
Cache-based test-time adaptation improves CLIP predictions by storing and retrieving examples from the target stream while keeping the model frozen. However, existing methods largely treat the feature space used for image-image retrieval as fixed, leaving open how much adaptation depends on the retrieval space itself. We study this question by fixing the memory and changing only the retrieval encoder, finding that the same memory can yield very different gains: across sixteen retrieval spaces, ImageNet-A cache gain ranges from at most +0.44 points for CLIP and MAE to +19.7 +/- 0.4 for DINOv2-L, while label-free retrieval-space selection retains 98% of oracle gain on ImageNet-V2. These results show that memory quality depends not only on which examples are stored, but also on how they are retrieved. Motivated by this finding, we propose MARC (Memory Augmented Retrieval for CLIP), a training-free system that uses frozen CLIP for prediction and DINOv2-B for retrieval with a single fusion weight. A single-view cache repairs 1074 +/- 21 baseline errors, compared with 878 +/- 4 for a 64-view ensemble, at roughly one seventh of the cost. Across four ImageNet distribution shifts, MARC reaches a 67.91% OOD average and, at matched DINOv2-B scale and eight views, achieves 64.17 +/- 0.31% versus 62.75 +/- 0.15% for a graph-based cache system while running 2.6 times faster. Overall, our results establish retrieval space as a first-order design choice for robust cache-based adaptation in remote sensing, scientific imaging, and changing visual environments.
发表机构
- North South University（南北大学）
- Jahangirnagar University（贾汉吉尔纳加尔大学）
- Tsinghua University（清华大学）
- Charles Sturt University（查尔斯特大学）
机构由 AI 辅助整理，请以论文原文为准。
