arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2002.08910cs.CLcs.LGstat.ML
收藏

语言模型的参数中可以包含多少知识?

How Much Knowledge Can You Pack Into the Parameters of a Language Model?

Google(谷歌)

Adam Roberts, Colin Raffel, Noam Shazeer

摘要:

最近已经观察到,在非结构化文本上训练的神经语言模型可以使用自然语言查询隐式地存储和检索知识。在这篇简短的论文中,我们通过微调预训练模型来衡量这种方法的实际效用,以在不访问任何外部背景或知识的情况下回答问题。我们表明,这种方法的规模与模型的大小,并执行竞争力与开放域系统,明确地从外部知识源检索答案时回答问题。为了促进可重复性和未来的工作,我们在https://goo.gle/t5-cbqa上发布了我们的代码和训练模型。

英文摘要:

It has recently been observed that neural language models trained on unstructured text can implicitly store and retrieve knowledge using natural language queries. In this short paper, we measure the practical utility of this approach by fine-tuning pre-trained models to answer questions without access to any external context or knowledge. We show that this approach scales with model size and performs competitively with open-domain systems that explicitly retrieve answers from an external knowledge source when answering questions. To facilitate reproducibility and future work, we release our code and trained models at https://goo.gle/t5-cbqa.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑