arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.09855cs.CLcs.AI
收藏

构建INDIC语言的预训练大语言模型数据集:以印地语为案例研究

Building pre-train LLM Dataset for the INDIC Languages: a case study on Hindi

Silo AI(西洛人工智能公司);Banasthali Vidyapith(班斯塔利维迪亚皮斯大学);Sharda University(沙尔达大学);Vellore Institute of Technology(韦洛尔理工学院);Odia Generative AI(奥迪亚生成式人工智能公司)

Shantipriya Parida, Shakshi Panwar, Kusum Lata, Sanskruti Mishra, Sambit Sekhar

AI总结:

针对Indic语言大语言模型构建缺乏高质量预训练数据的问题,本文构建了含12.8亿token、覆盖多领域及主要方言的印地语预训练数据集,阐述了数据处理流程,方法可扩展至其他低资源语言且将免费开放。

摘要:

大语言模型(LLM)在众多需要根据人类指令自动生成回复的应用中展现出了变革性能力。然而,构建大语言模型(尤其是针对Indic语言的模型)的主要挑战在于缺乏用于构建基础大语言模型的高质量数据。本文提出了一个适用于Indic语言印地语的大型预训练数据集。我们收集了涵盖多个领域的数据,包括印地语的主要方言。该数据集包含12.8亿个印地语token。我们阐述了包括数据收集、预处理以及大语言模型预训练可用性在内的完整流程。所提出的方法可轻松扩展至其他Indic语言及低资源语言,且将免费开放用于大语言模型预训练及相关研究。

英文摘要:

Large language models (LLMs) demonstrated transformative capabilities in many applications that require automatically generating responses based on human instruction. However, the major challenge for building LLMs, particularly in Indic languages, is the availability of high-quality data for building foundation LLMs. In this paper, we are proposing a large pre-train dataset in Hindi useful for the Indic language Hindi. We have collected the data span across several domains including major dialects in Hindi. The dataset contains 1.28 billion Hindi tokens. We have explained our pipeline including data collection, pre-processing, and availability for LLM pre-training. The proposed approach can be easily extended to other Indic and low-resource languages and will be available freely for LLM pre-training and LLM research purposes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑