arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.17832cs.CL
收藏

波兰语分类任务中少样本学习的评估

Evaluation of Few-Shot Learning for Classification Tasks in the Polish Language

ML Research at Allegro(Allegro公司机器学习研究部门)

Tsimur Hadeliya, Dariusz Kajtoch

AI总结:

本文引入了包含7个波兰语原生分类任务的少样本基准,通过对比微调、线性探测、SetFit和上下文学习等方法,发现ICL表现最佳,但与全量数据微调仍存在14个百分点差距。

摘要:

我们引入了一个由7个波兰语原生分类任务组成的少样本基准。我们使用各种预训练的商业和开源模型,在0样本和16样本下对微调、线性探测、SetFit和上下文学习(ICL)进行了实证比较。研究结果表明,ICL取得了最佳性能,其中GPT-3.5和GPT-4等商业模型表现最优。然而,我们最佳的少样本学习得分与HerBERT-large在整个训练数据集上微调的性能之间仍存在14个百分点的显著差距。在这些技术中,SetFit是第二好的方法,紧随其后的是线性探测。我们观察到非线性头微调的性能最差且最不稳定。ICL的结果表明,在波兰语语料库上对Mistral-7b或Llama-2-13b等模型进行持续预训练是有益的。Bielik-7b和Trurl-13b的性能提升分别证实了这一点。为了进一步支持波兰语的少样本学习实验,我们发布了手工制作的ICL模板。

英文摘要:

We introduce a few-shot benchmark consisting of 7 different classification tasks native to the Polish language. We conducted an empirical comparison with 0 and 16 shots between fine-tuning, linear probing, SetFit, and in-context learning (ICL) using various pre-trained commercial and open-source models. Our findings reveal that ICL achieves the best performance, with commercial models like GPT-3.5 and GPT-4 attaining the best performance. However, there remains a significant 14 percentage points gap between our best few-shot learning score and the performance of HerBERT-large fine-tuned on the entire training dataset. Among the techniques, SetFit emerges as the second-best approach, closely followed by linear probing. We observed the worst and most unstable performance with non-linear head fine-tuning. Results for ICL indicate that continual pre-training of models like Mistral-7b or Llama-2-13b on Polish corpora is beneficial. This is confirmed by the improved performances of Bielik-7b and Trurl-13b, respectively. To further support experiments in few-shot learning for Polish, we are releasing handcrafted templates for the ICL.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑