IndicLLMSuite：创建印度语言预训练与微调数据集的蓝图
IndicLLMSuite: A Blueprint for Creating Pre-training and Fine-Tuning Datasets for Indian Languages
AI总结：
本研究针对印度语言LLM资源稀缺问题，构建覆盖22种语言的2510亿token预训练数据与7480万指令对微调数据套件，整合多类型数据并提供开源管道，助力印度语言LLM发展并为其他语言提供参考蓝图。
摘要：
尽管英文大语言模型（LLMs）取得了显著进展，但由于缺乏定制化资源，构建其他语言可比模型的进展受阻。本研究旨在通过引入一套专门为印度语言LLM开发设计的扩展资源套件来弥合这一差距，该套件覆盖22种语言，包含总计2510亿个token和7480万条指令-响应对。认识到数据质量和数量的重要性，我们的方法结合了高度精选的人工验证数据、未经验证但有价值的数据以及合成数据。我们构建了一个清洁的开源管道，用于从网站、PDF和视频等多样化来源中筛选预训练数据，整合了爬取、清理、标记和去重的最佳实践。对于指令微调，我们整合了现有的印度语言数据集，将英文数据集翻译/音译为印度语言，并利用LLaMa2和Mixtral模型创建基于印度维基百科和Wikihow文章的对话。此外，我们通过为多种场景生成有毒提示词，然后将这些有毒提示词输入对齐的LLaMa2模型生成无毒响应，以解决毒性对齐问题。我们希望作为本研究一部分发布的数据集、工具和资源，不仅能推动印度语言LLM的研究与开发，还能为将此类工作扩展到其他语言建立开源蓝图。本研究创建的数据和其他成果以宽松许可证发布。
英文摘要：
Despite the considerable advancements in English LLMs, the progress in building comparable models for other languages has been hindered due to the scarcity of tailored resources. Our work aims to bridge this divide by introducing an expansive suite of resources specifically designed for the development of Indic LLMs, covering 22 languages, containing a total of 251B tokens and 74.8M instruction-response pairs. Recognizing the importance of both data quality and quantity, our approach combines highly curated manually verified data, unverified yet valuable data, and synthetic data. We build a clean, open-source pipeline for curating pre-training data from diverse sources, including websites, PDFs, and videos, incorporating best practices for crawling, cleaning, flagging, and deduplication. For instruction-fine tuning, we amalgamate existing Indic datasets, translate/transliterate English datasets into Indian languages, and utilize LLaMa2 and Mixtral models to create conversations grounded in articles from Indian Wikipedia and Wikihow. Additionally, we address toxicity alignment by generating toxic prompts for multiple scenarios and then generate non-toxic responses by feeding these toxic prompts to an aligned LLaMa2 model. We hope that the datasets, tools, and resources released as a part of this work will not only propel the research and development of Indic LLMs but also establish an open-source blueprint for extending such efforts to other languages. The data and other artifacts created as part of this work are released with permissive licenses.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
