Retrieval-augmented language models (LMs) use non-parametric memory to substantially outperform their non-retrieval counterparts on perplexity-based evaluations, but it is an open question whether they achieve similar gains in few- and zero-shot end-task accuracy. We extensively study one such model, the k-nearest neighbor LM (kNN-LM), showing that the gains marginally transfer. The main challenge is to achieve coverage of the verbalizer tokens that define the different end-task class labels. To address this challenge, we also introduce kNN-Prompt, a simple and effective kNN-LM with automatically expanded fuzzy verbalizers (e.g. to expand terrible to also include silly and other task-specific synonyms for sentiment classification). Across nine diverse end-tasks, using kNN-Prompt with GPT-2 large yields significant performance boosts over strong zero-shot baselines (13.4% absolute improvement over the base LM on average). We also show that other advantages of non-parametric augmentation hold for end tasks; kNN-Prompt is effective for domain adaptation with no further training, and gains increase with the size of the retrieval model.
kNN-ICL: Compositional Task-Oriented Parsing Generalization with Nearest Neighbor In-Context Learning
University of Illinois Chicago(伊利诺伊大学芝加哥分校);Salesforce Research(赛富时研究院);Huazhong University of Science and Technology(华中科技大学);Columbia University(哥伦比亚大学);The Hong Kong Polytechnic University(香港理工大学);State University of New York at Binghamton(纽约州立大学宾汉姆顿分校)
BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA
Center for Information and Language Processing (CIS)(信息与语言处理中心(CIS));LMU Munich(慕尼黑大学)
Khandelwal等人。(2020)使用k最近邻(kNN)组件来提高语言模型性能。我们表明,这种想法是有益的开放域的问答(QA)。为了提高对训练过程中遇到的事实的回忆,我们结合了BERT(Devlin et al.,2019)与传统的信息检索步骤(IR)和在嵌入式文本集合的大范围上的kNN搜索。我们的贡献如下:i)BERT-kNN在完形填空式问答中的性能大大优于BERT,无需任何进一步的训练。ii)我们表明,BERT经常识别正确的响应类别(例如,美国城市),但只有kNN恢复事实上正确的答案(例如,"Miami")。iii)与BERT相比,BERT-kNN在罕见事实方面表现出色。iv)BERT-kNN可以很容易地处理BERT的训练集没有覆盖的事实,例如,最近的事件。