适配器能写入多少比特？测量参数高效微调的容量和记忆能力
How Much Can a LoRA Adapter Memorize? Measuring Adapter Capacity in Bits
浏览论文内容
中文总结 AI 辅助
研究探讨 LoRA 适配器能写入多少比特，通过扩展记忆分析直接测量其写入模型的信息量，发现写入量小于完全微调且不规律，受参数位置影响，还用于分析 Qwen2.5 微调中的隐私泄露，区分不同学习方式，将经验转化为可防范量。
中文摘要 AI 辅助
LoRA 适配器虽仅几兆字节，却常被视为一种技能而非数据记录。我们对此假设进行评估，将基于压缩的记忆分析扩展到冻结基础设置，直接以比特为单位测量低秩适配器写入模型且从不改变的信息量。结果表明，其写入量小于完全微调，且不像参数计数预测的那样规律。适配器每个可训练参数存储几位信息，远低于完整模型的预算，关键在于参数所在位置而非数量。应用于 Qwen2.5 的实际微调，该方法表明隐私泄露随适配器写入的比特数增加，而非名义上的参数数增加，且能区分监督学习和强化学习。测量微调写入的内容，而非事后攻击，能将一种经验法则转化为可设计防范的量。
英文摘要
LoRA adapters are often shared on their own, and the amount of information they can hold about their training data bounds what sharing them can reveal. Following Morris et al. (2026), we measure this amount in bits by training adapters on frozen language models to memorize random token sequences. LoRA adapters store 2 to 3.4 bits per trainable parameter, less than full fine-tuning of the same model. The number of parameters alone does not set this amount: at equal size, adapters on MLP layers store more than adapters on attention layers, and a randomly initialized frozen model supports as much storage as a pretrained one once the scale of its output logits can be trained. We then train the same adapter on a task with supervised fine-tuning (SFT) and with GRPO. At the same accuracy, SFT stores about three times as much information specific to its training examples, and it stores most of the bits of secrets planted in the training data, which GRPO does not store because it rarely samples them.
