Pre-trained language models(PLM) have made impressive results in various NLP tasks. It has been revealed that one of the key factors to their success is the parameters of these models implicitly learn all kinds of knowledge during pre-training. However, encoding knowledge implicitly in the model parameters has two fundamental drawbacks. First, the knowledge is neither editable nor scalable once the model is trained, which is especially problematic in that knowledge is consistently evolving. Second, it lacks interpretability and prevents humans from understanding which knowledge PLM requires for a certain problem. In this paper, we introduce PlugLM, a pre-training model with differentiable plug-in memory(DPM). The key intuition is to decouple the knowledge storage from model parameters with an editable and scalable key-value memory and leverage knowledge in an explainable manner by knowledge retrieval in the DPM. To justify this design choice, we conduct evaluations in three settings including: (1) domain adaptation. PlugLM obtains 3.95 F1 improvements across four domains on average without any in-domain pre-training. (2) knowledge update. PlugLM could absorb new knowledge in a training-free way after pre-training is done. (3) in-task knowledge learning. PlugLM could be further improved by incorporating training samples into DPM with knowledge prompting.
Decoupled Alignment for Robust Plug-and-Play Adaptation
Northwestern University(西北大学);New York University Abu Dhabi(纽约大学阿布扎克分校);Stanford University(斯坦福大学);Texas A&M University(德克萨斯农工大学);University of California, Los Angeles(加州大学洛杉矶分校);Illinois Institute of Technology(伊利诺伊理工学院)
Variator: Accelerating Pre-trained Models with Plug-and-Play Compression Modules
Tsinghua University(清华大学);Quan Cheng Laboratory(泉城实验室);Zhili College, Tsinghua University(清华大学志道书院);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Tencent Inc.(腾讯公司)
Plug-and-Play Knowledge Injection for Pre-trained Language Models
Renmin University of China(中国人民大学);Tsinghua University(清华大学);International Innovation Center of Tsinghua University(清华大学国际创新中心);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));Tencent Inc(腾讯公司);Quan Cheng Laboratory(泉城实验室)
Plug-and-Play Document Modules for Pre-trained Models
Tsinghua University(清华大学);International Innovation Center of Tsinghua University, Shanghai(清华大学上海国际创新中心);Quan Cheng Laboratory(之江实验室);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Beijing Key Laboratory of Big Data Management and Analysis Methods(大数据管理与分析方法北京市重点实验室);Huawei Technologies Co., Ltd.(华为技术有限公司)