Novel-WD:利用前缀微调探索大语言模型获取新世界知识
Novel-WD: Exploring acquisition of Novel World Knowledge in LLMs Using Prefix-Tuning
AI总结:
本文提出Novel-WD数据集以研究大语言模型获取预训练语料之外新世界知识的能力,并验证前缀微调可有效存储新事实,且容量随前缀长度和模型规模增长。
摘要:
向预训练大语言模型(PLM)传授新信息是一项至关重要但具有挑战性的任务。模型适配技术(如微调和参数高效训练)已被证明存储新事实的速度较慢;持续学习是一种可选方案,但成本高昂且容易发生灾难性遗忘。本研究探讨并量化了PLM如何学习并记住未出现在其预训练语料库中的新世界知识事实,因为预训练语料库仅包含截至某一日期的世界知识。为此,我们首先提出了Novel-WD,这是一个新数据集,由从近期Wikidata更新中提取的包含新事实的句子组成,并配套两个评估任务,形式为因果语言建模和多项选择题(MCQ)。我们将该数据集免费提供给社区,并发布了一套流程,以便后续利用最新信息构建类似数据集的新版本。我们还探索了使用前缀微调进行新信息学习,并分析了在给定前缀中可以存储多少信息。我们表明,单个事实可以可靠地编码在单个前缀中,并且前缀容量随其长度和基础模型规模的增加而增加。
英文摘要:
Teaching new information to pre-trained large language models (PLM) is a crucial but challenging task. Model adaptation techniques, such as fine-tuning and parameter-efficient training have been shown to store new facts at a slow rate; continual learning is an option but is costly and prone to catastrophic forgetting. This work studies and quantifies how PLM may learn and remember new world knowledge facts that do not occur in their pre-training corpus, which only contains world knowledge up to a certain date. To that purpose, we first propose Novel-WD, a new dataset consisting of sentences containing novel facts extracted from recent Wikidata updates, along with two evaluation tasks in the form of causal language modeling and multiple choice questions (MCQ). We make this dataset freely available to the community, and release a procedure to later build new versions of similar datasets with up-to-date information. We also explore the use of prefix-tuning for novel information learning, and analyze how much information can be stored within a given prefix. We show that a single fact can reliably be encoded within a single prefix, and that the prefix capacity increases with its length and with the base model size.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。