arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.17532cs.CL
收藏

检索即精准生成

Retrieval is Accurate Generation

School of ECE, Peking University(北京大学电子学院);Tencent AI Lab(腾讯人工智能实验室)

Bowen Cao, Deng Cai, Leyang Cui, Xuxin Cheng, Wei Bi, Yuexian Zou, Shuming Shi

AI总结:

本文提出从支持文档中选取上下文感知短语的检索式生成方法,通过语言启发式初始化训练预言并迭代自强化,在知识密集型任务和开放式生成上显著提升性能,实现更精准的生成。

摘要:

标准语言模型通过从固定、有限且独立的词汇表中选取词元来生成文本。我们引入了一种新方法,从一组支持文档中选取上下文感知的短语。这种范式转变面临的最大挑战之一是确定训练预言,因为一段文本可以有多种切分方式,且每个片段可以从众多可能的文档中检索得到。为解决此问题,我们提出使用语言启发式初始化训练预言,更重要的是,通过迭代自强化对预言进行引导。大量实验表明,我们的模型不仅在多种知识密集型任务上优于标准语言模型,而且在开放式文本生成中展现出更优的生成质量。例如,与标准语言模型对应物相比,我们的模型在OpenbookQA上的准确率从23.47%提升至36.27%,在开放式文本生成中的MAUVE分数从42.61%提升至81.58%。值得注意的是,我们的模型还在多个检索增强基线中实现了最佳性能和最低延迟。总之,我们断言检索即更精准的生成,并希望我们的工作能鼓励对这一新范式转变的进一步研究。

英文摘要:

Standard language models generate text by selecting tokens from a fixed, finite, and standalone vocabulary. We introduce a novel method that selects context-aware phrases from a collection of supporting documents. One of the most significant challenges for this paradigm shift is determining the training oracles, because a string of text can be segmented in various ways and each segment can be retrieved from numerous possible documents. To address this, we propose to initialize the training oracles using linguistic heuristics and, more importantly, bootstrap the oracles through iterative self-reinforcement. Extensive experiments show that our model not only outperforms standard language models on a variety of knowledge-intensive tasks but also demonstrates improved generation quality in open-ended text generation. For instance, compared to the standard language model counterpart, our model raises the accuracy from 23.47% to 36.27% on OpenbookQA, and improves the MAUVE score from 42.61% to 81.58% in open-ended text generation. Remarkably, our model also achieves the best performance and the lowest latency among several retrieval-augmented baselines. In conclusion, we assert that retrieval is more accurate generation and hope that our work will encourage further research on this new paradigm shift.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑