arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.04581cs.CLcs.AIcs.LG
收藏

基于QA记忆的开放域问答语言模型扩充

Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answering

Google Research(谷歌研究院);University of Southern California(南加州大学)

Wenhu Chen, Pat Verga, Michiel de Jong, John Wieting, William Cohen

摘要:

检索增强语言模型最近已成为知识密集型任务的标准。这些方法不是纯粹依赖于大型神经模型参数中的潜在语义,而是使用半参数记忆来编码模型检索的知识索引。大多数先前的工作采用文本段落作为知识的单位,其具有高覆盖率,但代价是可解释性,可控性和效率。相反的性质出现在其他方法,而不是依赖于知识库(KB)的事实。与此同时,最近的工作已经证明了存储和检索从文本\citep{lewis2021paq}派生的Q-A对的索引的有效性。这种方法产生了一个高覆盖率的知识表示,保持知识库一样的属性,由于其表示是更多的原子单元的信息。在这项工作中,我们通过提出一个问答增强的编码器-解码器模型和伴随的预训练策略进一步推动了这一研究。这产生了一个端到端的系统,不仅优于以前的QA检索方法的单跳QA任务,但也使组合推理,表现出强大的性能在两个多跳QA数据集。总之,这些方法提高了解释和控制模型的能力,同时缩小了与段落检索系统的性能差距。

英文摘要:

Retrieval augmented language models have recently become the standard for knowledge intensive tasks. Rather than relying purely on latent semantics within the parameters of large neural models, these methods enlist a semi-parametric memory to encode an index of knowledge for the model to retrieve over. Most prior work has employed text passages as the unit of knowledge, which has high coverage at the cost of interpretability, controllability, and efficiency. The opposite properties arise in other methods which have instead relied on knowledge base (KB) facts. At the same time, more recent work has demonstrated the effectiveness of storing and retrieving from an index of Q-A pairs derived from text \citep{lewis2021paq}. This approach yields a high coverage knowledge representation that maintains KB-like properties due to its representations being more atomic units of information. In this work we push this line of research further by proposing a question-answer augmented encoder-decoder model and accompanying pretraining strategy. This yields an end-to-end system that not only outperforms prior QA retrieval methods on single-hop QA tasks but also enables compositional reasoning, as demonstrated by strong performance on two multi-hop QA datasets. Together, these methods improve the ability to interpret and control the model while narrowing the performance gap with passage retrieval systems.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑