Pragyaan：为印度语言设计与策展高质量文化后训练数据集
Pragyaan: Designing and Curating High-Quality Cultural Post-Training Datasets for Indian Languages
AI总结：
针对印度语言后训练数据缺乏多语言覆盖与文化根基的问题，提出人机协同流水线结合翻译与合成扩展，构建Pragyaan-IT和Pragyaan-Align数据集，强调任务多样性与文化保留，以提升多语言LLM的包容性和有效性。
摘要：
大型语言模型（LLM）的有效性在很大程度上取决于高质量后训练数据的可用性，特别是指令微调和基于偏好的示例。然而，现有的开源数据集往往缺乏多语言覆盖、文化根基，并且存在任务多样性差距，这些问题在印度语言中尤为突出。我们引入了一种人机协同（human-in-the-loop）流水线，将翻译与合成扩展相结合，以生成可靠且多样化的印度语后训练数据。利用该流水线，我们策展了两个数据集：Pragyaan-IT（22.5K）和Pragyaan-Align（100K），涵盖10种印度语言，覆盖13个大类和56个子类别，利用了57个多样化数据集。我们的数据集协议纳入了若干常被忽视的维度，并强调任务多样性、多轮对话、指令保真度、安全对齐以及文化细微差别的保留，为更具包容性和有效性的多语言LLM奠定了基础。
英文摘要：
The effectiveness of Large Language Models (LLMs) depends heavily on the availability of high-quality post-training data, particularly instruction-tuning and preference-based examples. Existing open-source datasets, however, often lack multilingual coverage, cultural grounding, and suffer from task diversity gaps that are especially pronounced for Indian languages. We introduce a human-in-the-loop pipeline that combines translations with synthetic expansion to produce reliable and diverse Indic post-training data. Using this pipeline, we curate two datasets: Pragyaan-IT (22.5K) and Pragyaan-Align (100K) across 10 Indian languages covering 13 broad and 56 sub-categories, leveraging 57 diverse datasets. Our dataset protocol incorporates several often-overlooked dimensions and emphasize task diversity, multi-turn dialogue, instruction fidelity, safety alignment, and preservation of cultural nuance, providing a foundation for more inclusive and effective multilingual LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
