CUTE：一个用于增强低资源语言跨语言知识迁移的多语言数据集
CUTE: A Multilingual Dataset for Enhancing Cross-Lingual Knowledge Transfer in Low-Resource Languages
AI总结：
本文构建并开源了包含中、英、维吾尔、藏四语的大规模语料库CUTE，验证了其机器翻译质量，并证明该数据集能有效提升大语言模型对低资源语言的处理能力。
摘要：
大型语言模型（LLMs）在各种自然语言处理任务中展现出卓越的零样本能力，显著提升了用户体验和效率。然而，这一优势主要局限于资源丰富的语言。对于种类繁多的低资源语言，支持仍然不足，训练语料的稀缺被认为是主要原因。我们构建并开源了CUTE中文、维吾尔文、藏文、英文数据集，该数据集由两个25GB的四语言语料集合组成（一个平行语料集和一个非平行语料集），通过机器翻译获得。CUTE涵盖两种资源丰富的语言（中文和英文）和两种低资源语言（维吾尔文和藏文）。在构建CUTE之前，人工评估验证了中文-维吾尔文和中文-藏文之间的机器翻译质量接近中文-英文翻译的质量。CUTE是迄今为止最大的维吾尔文和藏文开源语料库，我们证明了其在增强LLMs处理低资源语言能力方面的有效性，同时研究了语料平行性在跨语言迁移学习中的作用。CUTE语料库及相关模型已向研究社区公开提供。
英文摘要：
Large Language Models (LLMs) demonstrate exceptional zero-shot capabilities in various NLP tasks, significantly enhancing user experience and efficiency. However, this advantage is primarily limited to resource-rich languages. For the diverse array of low-resource languages, support remains inadequate, with the scarcity of training corpora considered the primary cause. We construct and open-source CUTE Chinese, Uyghur, Tibetan,English dataset, consisting of two 25GB sets of four-language corpora (one parallel and one non-parallel), obtained through machine translation. CUTE encompasses two resource-rich languages (Chinese and English) and two low-resource languages (Uyghur and Tibetan). Prior to constructing CUTE, human assessment validates that the machine translation quality between Chinese-Uyghur and Chinese-Tibetan approaches that of Chinese-English translation. CUTE represents the largest open-source corpus for Uyghur and Tibetan languages to date, and we demonstrate its effectiveness in enhancing LLMs' ability to process low-resource languages while investigating the role of corpus parallelism in cross-lingual transfer learning. The CUTE corpus and related models are made publicly available to the research community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
