面向低资源语言大语言模型提示的少样本跨语言迁移
Few-Shot Cross-Lingual Transfer for Prompting Large Language Models in Low-Resource Languages
AI总结：
针对低资源语言大模型提示能力不足的问题，该研究对比少样本提示、语言自适应微调、机器翻译三种方法，在三类任务上验证少样本提示的平均效果最优且更具成本效益。
摘要：
大型预训练语言模型（PLMs）处于自然语言处理进展的前沿。PLMs的一个广泛应用场景是“提示”（即上下文学习）：用户先向PLM提供任务描述和若干已完成的任务示例作为上下文，再提示PLM对新示例执行该任务。只有规模最大、能力最强的PLMs才能有效执行上下文学习，而这些模型通常以英语语料为主进行训练，导致其他语言的发展滞后。大多数语言的数据局限性使得无法训练出支持提示功能的单语言PLMs。尽管提示设置相关的研究工作激增，但目前仍不清楚应如何针对提示任务对PLMs进行跨语言适配。\n我们评估了将LLaMa（一款拥有7B参数、主要以英语训练的开源PLM）适配为低资源语言（即卢旺达语、豪萨语和卢干达语）提示模型的可行方法。我们研究了三种方法：少样本提示（prompt）、语言自适应微调（LAFT）和神经机器翻译（translate），并在抽象式摘要、多类别主题分类和命名实体识别任务上进行评估。尽管LAFT的计算成本最高，且直观上应能带来最佳结果，但我们的实验表明，在适配PLMs以用于提示任务时，LAFT仅在少数情况下是最优选择。相反，对于选定的低资源语言，translate和prompt设置是计算高效且成本划算的少样本提示方法。我们发现结果具有任务和语言依赖性，但在所有任务和语言的平均表现上，prompt方法表现最佳。结果显示，在所有任务和语言上聚合统计时，对于所有少样本数量设置，prompt设置的表现均优于translate和LAFT，且具有统计显著性。
英文摘要：
Large pre-trained language models (PLMs) are at the forefront of advances in Natural Language Processing. One widespread use case of PLMs is "prompting" - or in-context learning - where a user provides a description of a task and some completed examples of the task to a PLM as context before prompting the PLM to perform the task on a new example. Only the largest, most capable PLMs are able to perform in-context learning effectively, and these models are typically trained with a predominantly English corpus, leaving all other languages behind. The data limitations in most languages preclude the training of language-specific PLMs capable of prompting. Albeit the surge in work of prompting settings, it is still unclear how PLMs should be adapted cross-lingually specifically for prompting. We evaluate the possible methods to adapt LLaMa, a 7B parameter open-source PLM mainly trained in English, for prompting in low-resource languages, namely for Kinyarwanda, Hausa, and Luganda. We consider three methods: few-shot prompting (prompt), language-adaptive fine-tuning (LAFT), and neural machine translation (translate), and evaluate on abstractive summarization, multi-class topic classification, and named-entity recognition. Although LAFT carries the greatest compute cost and intuitively should lead to the best results, our experiments exhibit that LAFT is only occasionally the optimal choice for adapting PLMs for prompting. Rather, the translate and prompt settings are a compute-efficient and cost-effective method of few-shot prompting for the selected low-resource languages. We find that the results are task and language dependent but find that the prompting method is the best on average across all tasks and languages. Results show that the prompt setting performs better than both translating and LAFT with statistical significance for all shots when aggregated across all tasks and languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
