arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2304.09058cs.CLcs.AIcs.DBcs.IRcs.LG
收藏

重新审视k-NN用于预训练语言模型的微调

Revisiting k-NN for Fine-tuning Pre-trained Language Models

Zhejiang University(浙江大学);AZFT Joint Lab for Knowledge Engine(AZFT知识引擎联合实验室)

Lei Li, Jing Chen, Bozhong Tian, Ningyu Zhang

AI总结:

本文重新审视kNN分类器以增强预训练语言模型,通过kNN校准训练和概率插值,在多种设置和任务上验证了经典方法的有效性。

摘要:

预训练语言模型(PLMs)作为基于参数的急切学习器,已成为当前自然语言处理(NLP)范式的默认选择。相比之下,k-最近邻(kNN)分类器作为惰性学习范式,倾向于缓解过拟合和孤立噪声。在本文中,我们重新审视kNN分类器以增强基于PLMs的分类器。在方法论层面,我们提出采用kNN与PLMs的文本表示相结合的两步方法:(1)利用kNN作为先验知识来校准训练过程。(2)将kNN预测的概率分布与PLMs分类器的概率分布进行线性插值。我们方法的核心是kNN校准训练的实现,该训练将预测结果视为训练过程中简单与困难样本的指标。从应用场景多样性的角度,我们在微调、提示调优范式以及零样本、少样本和全监督设置下,分别对八个不同的端任务进行了广泛实验。我们希望我们的探索能鼓励社区重新审视经典方法在高效NLP中的力量。代码和数据集可在https://github.com/zjunlp/Revisit-KNN获取。

英文摘要:

Pre-trained Language Models (PLMs), as parametric-based eager learners, have become the de-facto choice for current paradigms of Natural Language Processing (NLP). In contrast, k-Nearest-Neighbor (kNN) classifiers, as the lazy learning paradigm, tend to mitigate over-fitting and isolated noise. In this paper, we revisit kNN classifiers for augmenting the PLMs-based classifiers. From the methodological level, we propose to adopt kNN with textual representations of PLMs in two steps: (1) Utilize kNN as prior knowledge to calibrate the training process. (2) Linearly interpolate the probability distribution predicted by kNN with that of the PLMs' classifier. At the heart of our approach is the implementation of kNN-calibrated training, which treats predicted results as indicators for easy versus hard examples during the training process. From the perspective of the diversity of application scenarios, we conduct extensive experiments on fine-tuning, prompt-tuning paradigms and zero-shot, few-shot and fully-supervised settings, respectively, across eight diverse end-tasks. We hope our exploration will encourage the community to revisit the power of classical methods for efficient NLP. Code and datasets are available in https://github.com/zjunlp/Revisit-KNN.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑