学习在视觉Transformer中的编辑位置
Learning Where to Edit Vision Transformers
AI总结：
本文提出一种定位-编辑方法，通过元学习超网络生成掩码识别ViT中需编辑的稀疏参数，并微调以纠正子群体偏移引起的预测错误，在构建的基准上实现泛化与局部性的可调权衡。
摘要：
模型编辑旨在以数据高效的方式纠正大型预训练模型的预测错误，同时确保对邻近失败的泛化能力以及局部性，以最小化对无关示例的意外影响。尽管在编辑基于Transformer的大型语言模型方面已取得显著进展，但在计算机视觉中有效编辑视觉Transformer（ViT）的策略仍 largely 未被开发。本文迈出了纠正ViT预测错误（尤其是由子群体偏移引起的错误）的初步步骤。采用“定位-编辑”方法，我们首先通过元学习一个在CutMix增强数据上训练的超网络来解决“编辑位置”的挑战，该数据是为编辑可靠性而生成的。这个训练好的超网络产生可泛化的二值掩码，识别出对真实世界失败样本有响应的结构化模型参数的稀疏子集。随后，我们通过简单地使用梯度下降的一种变体对识别出的参数进行微调来解决“如何编辑”的问题，以实现成功的编辑。为了验证我们的方法，我们构建了一个编辑基准，引入了向自然代表性不足的图像和AI生成图像的子群体偏移，从而揭示了预训练ViT在物体识别中的局限性。我们的方法不仅在我们提出的基准上取得了优越的性能，还允许在泛化与局部性之间进行可调节的权衡。我们的代码可在 https://github.com/hustyyq/Where-to-Edit 获取。
英文摘要：
Model editing aims to data-efficiently correct predictive errors of large pre-trained models while ensuring generalization to neighboring failures and locality to minimize unintended effects on unrelated examples. While significant progress has been made in editing Transformer-based large language models, effective strategies for editing vision Transformers (ViTs) in computer vision remain largely untapped. In this paper, we take initial steps towards correcting predictive errors of ViTs, particularly those arising from subpopulation shifts. Taking a locate-then-edit approach, we first address the where-to-edit challenge by meta-learning a hypernetwork on CutMix-augmented data generated for editing reliability. This trained hypernetwork produces generalizable binary masks that identify a sparse subset of structured model parameters, responsive to real-world failure samples. Afterward, we solve the how-to-edit problem by simply fine-tuning the identified parameters using a variant of gradient descent to achieve successful edits. To validate our method, we construct an editing benchmark that introduces subpopulation shifts towards natural underrepresented images and AI-generated images, thereby revealing the limitations of pre-trained ViTs for object recognition. Our approach not only achieves superior performance on the proposed benchmark but also allows for adjustable trade-offs between generalization and locality. Our code is available at https://github.com/hustyyq/Where-to-Edit.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
