鲁棒的权重签名:像修补权重一样容易获得鲁棒性?
Robust Weight Signatures: Gaining Robustness as Easy as Patching Weights?
摘要:
给定被训练为对一种或多种类型的分布偏移(例如,自然图像损坏),如何在模型权重中编码“鲁棒性”,以及如何容易地将其分离和/或“zero-shot”转移到某些其他模型?本文根据经验提出了一个令人惊讶的简单答案:线性-通过简单的模型权重算法!我们首先得出几个关键的观察结果:(1)假设我们在干净的数据集和其损坏的版本上训练相同的模型架构,得到的权重在浅层中大部分不同;(2)投影后的权重差异,我们称之为“鲁棒权重签名”(RWS),似乎是有区别的,并指示不同的损坏类型;(3)对于相同的腐败类型,同一模型架构得到的RWS具有高度一致性,并且可以在不同的数据集之间传递。我们提出了一个极简的模型鲁棒性“修补”框架,该框架携带一个在干净数据上训练的模型及其预提取的RWS。通过这种方式,向模型注入一定的鲁棒性被简化为直接将相应的RWS添加到其权重中。我们验证我们提出的框架是显着的(1)轻量级。由于RWS集中在最浅的几个层上,并且我们进一步表明它们可以被无痛地量化,因此存储RWS比存储全权重副本紧凑多达13倍;(2)原位可调。可以根据需要附加RWS,然后将其删除以恢复完整的干净模型。我们进一步证明了可以线性地重新缩放RWS来控制修补的鲁棒性强度;(3)可组合。可以同时添加多个RWS,以一次修补更全面的鲁棒性;以及(4)可转移。即使在不断调整或更新干净的模型主干时,RWS仍然是有效的补丁,因为它们具有出色的跨数据集可传输性。
英文摘要:
Given a robust model trained to be resilient to one or multiple types of distribution shifts (e.g., natural image corruptions), how is that "robustness" encoded in the model weights, and how easily can it be disentangled and/or "zero-shot" transferred to some other models? This paper empirically suggests a surprisingly simple answer: linearly - by straightforward model weight arithmetic! We start by drawing several key observations: (1)assuming that we train the same model architecture on both a clean dataset and its corrupted version, resultant weights mostly differ in shallow layers; (2)the weight difference after projection, which we call "Robust Weight Signature" (RWS), appears to be discriminative and indicative of different corruption types; (3)for the same corruption type, the RWSs obtained by one model architecture are highly consistent and transferable across different datasets. We propose a minimalistic model robustness "patching" framework that carries a model trained on clean data together with its pre-extracted RWSs. In this way, injecting certain robustness to the model is reduced to directly adding the corresponding RWS to its weight. We verify our proposed framework to be remarkably (1)lightweight. since RWSs concentrate on the shallowest few layers and we further show they can be painlessly quantized, storing an RWS is up to 13 x more compact than storing the full weight copy; (2)in-situ adjustable. RWSs can be appended as needed and later taken off to restore the intact clean model. We further demonstrate one can linearly re-scale the RWS to control the patched robustness strength; (3)composable. Multiple RWSs can be added simultaneously to patch more comprehensive robustness at once; and (4)transferable. Even when the clean model backbone is continually adapted or updated, RWSs remain as effective patches due to their outstanding cross-dataset transferability.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。