BUFFET：面向少样本跨语言迁移的大型语言模型基准测试
BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer
AI总结：
引入 BUFFET 基准统一 15 项任务和 54 种语言，评估多语言大模型在少样本跨语言迁移中的表现，发现上下文学习迁移效果仍不及微调的小模型。
摘要：
尽管自然语言处理在少样本泛化方面取得了显著进展，但大多数模型的开发和评估主要在英语上进行。为了促进少样本跨语言迁移的研究，我们引入了一个名为 BUFFET 的新基准，它以序列到序列的格式统一了 54 种语言中的 15 项不同任务，并提供了固定的少样本示例和指令集。BUFFET 旨在为跨广泛任务和语言的少样本跨语言迁移建立一个严格且公平的评估框架。使用 BUFFET，我们对最先进的多语言大型语言模型进行了不同迁移方法的全面评估，即上下文学习和微调。我们的发现表明，少样本上下文跨语言迁移存在巨大的改进空间。特别是，使用上下文学习的 ChatGPT 的表现通常比在英语任务数据和少样本同语言示例上微调的小得多的 mT5-base 模型更差。我们的分析为少样本跨语言迁移的未来研究提出了多种途径，例如改进预训练、理解和未来评估。
英文摘要：
Despite remarkable advancements in few-shot generalization in natural language processing, most models are developed and evaluated primarily in English. To facilitate research on few-shot cross-lingual transfer, we introduce a new benchmark, called BUFFET, which unifies 15 diverse tasks across 54 languages in a sequence-to-sequence format and provides a fixed set of few-shot examples and instructions. BUFFET is designed to establish a rigorous and equitable evaluation framework for few-shot cross-lingual transfer across a broad range of tasks and languages. Using BUFFET, we perform thorough evaluations of state-of-the-art multilingual large language models with different transfer methods, namely in-context learning and fine-tuning. Our findings reveal significant room for improvement in few-shot in-context cross-lingual transfer. In particular, ChatGPT with in-context learning often performs worse than much smaller mT5-base models fine-tuned on English task data and few-shot in-language examples. Our analysis suggests various avenues for future research in few-shot cross-lingual transfer, such as improved pretraining, understanding, and future evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
