PERL：在CLIP潜在空间中实现参数高效的推理
PERL: Parameter Efficient Reasoning in CLIP Latent Space
AI总结：
本文提出PERL，一种在CLIP潜在空间中通过迭代潜在推理实现参数高效适应的框架，该方法在多个基准测试中表现出最佳的参数-性能权衡，仅需约6K可训练参数即可实现强的新型类别准确率和竞争性的迁移性能。
摘要：
对比训练的视觉-语言模型，如CLIP，通过在共享嵌入空间中对齐图像和文本，提供了强大的零样本迁移能力。然而，将这些模型适应到下游任务而不影响其开放词汇泛化能力仍然具有挑战性。现有的参数高效适应方法通常通过学习的提示、适配器或多模态转换来提高任务专业化，其中适应能力主要通过额外的可训练参数来表达。受最近语言模型中潜在推理方法的启发，我们探讨了一种互补的视角：适应是否可以来自于对潜在表示的迭代推理，而不是仅仅通过增加参数数量？我们介绍了PERL（在CLIP潜在空间中实现参数高效的推理），一种轻量级的适应框架，它通过在冻结的CLIP模型上添加一个紧凑的共享推理模块，在多次细化步骤中反复应用。在每一步中，PERL根据当前的表示生成一个潜在推理标记，并将其注入到中间编码器层中，逐步细化更高层次的语义表示，同时保持CLIP的预训练多模态结构。在15个基准测试中，涵盖基础到新颖泛化、跨数据集迁移以及非分布ImageNet变体，PERL在快速适应的少样本设置下，实现了与其他方法相比最佳的参数-性能权衡，仅使用约6K可训练参数，比最大的比较方法少817倍，同时结合了强的新类别准确率和具有竞争力的迁移性能。总体而言，我们的结果表明，迭代的潜在推理为判别视觉-语言模型中的参数扩展提供了一种互补的适应机制。
英文摘要：
Contrastively trained vision-language models such as CLIP provide strong zero-shot transfer by aligning images and text in a shared embedding space. However, adapting these models to downstream tasks without degrading their open-vocabulary generalization remains challenging. Existing parameter-efficient adaptation methods typically improve task specialization through learned prompts, adapters, or multimodal transformations, where adaptation capacity is primarily expressed through additional trainable parameters. Inspired by recent latent reasoning methods in language models, we investigate a complementary perspective: can adaptation emerge from iterative reasoning on latent representations rather than from increasing parameter count alone? We introduce PERL (Parameter-Efficient Reasoning in CLIP Latent Space), a lightweight adaptation framework that augments a frozen CLIP model with a compact shared reasoning module applied recurrently across refinement steps. At each step, PERL generates a latent reasoning token conditioned on the current representation and injects it into an intermediate encoder layer, progressively refining higher-level semantic representations while preserving CLIP's pretrained multimodal structure. Across 15 benchmarks spanning base-to-novel generalization, cross-dataset transfer, and out-of-distribution ImageNet variants, PERL achieves the best parameter-performance trade-off among the compared methods under a fast-adaptation few-shot setting, combining strong novel-class accuracy and competitive transfer performance with only about 6K trainable parameters, up to 817x fewer than the largest compared approach. Overall, our results suggest that iterative latent reasoning provides a complementary adaptation mechanism to parameter scaling in discriminative vision-language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
