参数空间中的线性插值对微调语言模型足够好
Linear Interpolation In Parameter Space is Good Enough for Fine-Tuned Language Models
摘要：
在高维空间中获得两点之间连续插值的最简单方法是在它们之间画一条线。虽然以前的工作集中在模型参数之间的一般连接性上，但我们在微调后探索了预训练模型参数的线性插值。令人惊讶的是，我们可以在微调模型的中间点执行线性插值而不会降低性能。对于可控制的文本生成，这样的插值可以被视为朝向或抵靠期望的文本属性（例如，积极情绪），这可以用作用于可控文本生成而没有推理速度开销的进一步方法的基础。
英文摘要：
The simplest way to obtain continuous interpolation between two points in high dimensional space is to draw a line between them. While previous works focused on the general connectivity between model parameters, we explored linear interpolation for parameters of pre-trained models after fine-tuning. Surprisingly, we could perform linear interpolation without a performance drop in intermediate points for fine-tuned models. For controllable text generation, such interpolation could be seen as moving a model towards or against the desired text attribute (e.g., positive sentiment), which could be used as grounds for further methods for controllable text generation without inference speed overhead.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
