arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2205.13792cs.CL
收藏

kNN-Prompt:最近邻Zero-Shot推理

kNN-Prompt: Nearest Neighbor Zero-Shot Inference

Paul G. Allen School of Computer Science & Engineering(保罗·G·艾伦计算机科学与工程学院);University of Washington(华盛顿大学)

Weijia Shi, Julian Michael, Suchin Gururangan, Luke Zettlemoyer

摘要:

检索增强语言模型(LM)使用非参数记忆,大大优于他们的非检索同行的困惑为基础的评价,但它是一个悬而未决的问题,他们是否实现了类似的收益在几个和zero-shot最终任务的准确性。我们广泛地研究了这样一个模型,k-最近邻LM(kNN-LM),表明收益边际转移。主要的挑战是实现定义不同的最终任务类标签的动词化器标记的覆盖。为了应对这一挑战,我们还引入了kNN-Prompt,这是一个简单有效的kNN-LM,具有自动扩展的模糊动词(例如,扩展可怕的,也包括愚蠢和其他任务特定的同义词用于情感分类)。在九个不同的最终任务中,使用kNN提示与GPT-2大产生显着的性能提升超过强大的zero-shot基线(13.4%的绝对改善比基本LM平均)。我们还表明,其他优势的非参数增强持有终端任务; kNN提示是有效的领域自适应,没有进一步的训练,增益随着检索模型的大小。

英文摘要:

Retrieval-augmented language models (LMs) use non-parametric memory to substantially outperform their non-retrieval counterparts on perplexity-based evaluations, but it is an open question whether they achieve similar gains in few- and zero-shot end-task accuracy. We extensively study one such model, the k-nearest neighbor LM (kNN-LM), showing that the gains marginally transfer. The main challenge is to achieve coverage of the verbalizer tokens that define the different end-task class labels. To address this challenge, we also introduce kNN-Prompt, a simple and effective kNN-LM with automatically expanded fuzzy verbalizers (e.g. to expand terrible to also include silly and other task-specific synonyms for sentiment classification). Across nine diverse end-tasks, using kNN-Prompt with GPT-2 large yields significant performance boosts over strong zero-shot baselines (13.4% absolute improvement over the base LM on average). We also show that other advantages of non-parametric augmentation hold for end tasks; kNN-Prompt is effective for domain adaptation with no further training, and gains increase with the size of the retrieval model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑