Proxy-FDA：用于微调视觉基础模型而不遗忘的基于代理的特征分布对齐方法
Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting
AI总结：
提出Proxy-FDA正则化方法，通过最近邻图和信息代理在预训练与微调特征空间间进行特征分布对齐，显式保留结构知识，显著减少视觉基础模型微调中的概念遗忘。
摘要：
在海量数据上预训练的视觉基础模型编码了现实世界概念的丰富表示，可以通过微调适应下游任务。然而，在一个任务上微调基础模型往往会导致在其他任务上出现概念遗忘的问题。最近的鲁棒微调方法旨在减轻对先验知识的遗忘，同时不影响微调性能。通常通过匹配原始模型和微调模型的权重或特征对来保留知识。然而，这种逐点匹配可能过于严格，缺乏对同样编码丰富知识的特征邻域结构的显式感知。我们提出了一种新颖的正则化方法Proxy-FDA，它显式地保留了特征空间中的结构知识。Proxy-FDA在预训练和微调的特征空间之间执行特征分布对齐（使用最近邻图），并通过动态生成的信息代理来进一步提高对齐效果，这些代理旨在增加数据多样性。实验表明，Proxy-FDA显著减少了微调过程中的概念遗忘，并且我们发现遗忘与一种分布距离度量之间存在强相关性（相比于L2距离）。我们进一步展示了Proxy-FDA在各种微调设置（端到端、少样本和持续微调）以及图像分类、图像描述和VQA等不同任务中的优势。
英文摘要：
Vision foundation models pre-trained on massive data encode rich representations of real-world concepts, which can be adapted to downstream tasks by fine-tuning. However, fine-tuning foundation models on one task often leads to the issue of concept forgetting on other tasks. Recent methods of robust fine-tuning aim to mitigate forgetting of prior knowledge without affecting the fine-tuning performance. Knowledge is often preserved by matching the original and fine-tuned model weights or feature pairs. However, such point-wise matching can be too strong, without explicit awareness of the feature neighborhood structures that encode rich knowledge as well. We propose a novel regularization method Proxy-FDA that explicitly preserves the structural knowledge in feature space. Proxy-FDA performs Feature Distribution Alignment (using nearest neighbor graphs) between the pre-trained and fine-tuned feature spaces, and the alignment is further improved by informative proxies that are generated dynamically to increase data diversity. Experiments show that Proxy-FDA significantly reduces concept forgetting during fine-tuning, and we find a strong correlation between forgetting and a distributional distance metric (in comparison to L2 distance). We further demonstrate Proxy-FDA's benefits in various fine-tuning settings (end-to-end, few-shot and continual tuning) and across different tasks like image classification, captioning and VQA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
