Transformer前馈层中更新键值记忆的实证研究
Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers
AI总结:
本研究通过在多种大模型知识编辑和微调任务中进行实证消融对比,探究了更新Transformer前馈网络中键或值的不同影响,为理解FFNs的键值记忆机制提供了深刻见解。
摘要:
Transformer中的前馈网络(FFNs)被视为一组键值神经记忆,用于存储抽象的高层知识。在本研究中,我们对更新键(FFNs层中的第1层)或值(第2层)进行了实证消融研究。我们在大语言模型的各种知识编辑和微调任务中比较了这两种方法,以获取进一步理解FFNs的见解。代码可在$\href{https://github.com/qiuzh20/Tuning-keys-v.s.-values}{this\\,repo}$获取。
英文摘要:
The feed-forward networks (FFNs) in transformers are recognized as a group of key-value neural memories to restore abstract high-level knowledge. In this work, we conduct an empirical ablation study on updating keys (the 1st layer in the FFNs layer) or values (the 2nd layer in the FFNs layer). We compare those two methods in various knowledge editing and fine-tuning tasks of large language models to draw insights to understand FFNs further. Code is available at $\href{https://github.com/qiuzh20/Tuning-keys-v.s.-values}{this\,repo}$.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。