ks-lit-3m：一个310万词的克什米尔语文本数据集用于大语言模型预训练
ks-lit-3m: A 3.1 million word kashmiri text dataset for large language model pretraining
AI总结：
本文提出KS-LIT-3M数据集，通过开发InPage到Unicode转换器和严格预处理，解决克什米尔语预训练数据稀缺问题，为自然语言处理研究提供高质量资源。
摘要：
大语言模型（LLMs）在高资源语言上表现出色，但在克什米尔语上却无法生成连贯的文本。这种性能差异并非源于模型本身的限制，而是由于高质量训练数据的严重缺乏。数十年的克什米尔文学由于以专有InPage桌面出版格式存储，无法被现代NLP流程访问。本文介绍了KS-LIT-3M，一个经过精心挑选的310万词（1640万字符）数据集，专门用于预训练克什米尔语语言模型。该数据集结构为单一连续线性文本流，优化了因果语言模型训练，使模型能够从先前的上下文预测后续的token。该数据集通过开发专门的InPage到Unicode转换器，随后进行严格预处理，包括去除英语污染、字符规范化和质量验证。该数据集涵盖131607个独特的单词，来自文学作品、新闻写作、学术文本和宗教学术等不同领域，解决了克什米尔语技术的资源缺口。该数据集以CC-BY-4.0许可证发布，以促进克什米尔语自然语言处理的研究。
英文摘要：
Large Language Models (LLMs) demonstrate remarkable fluency across high-resource languages yet consistently fail to generate coherent text in Kashmiri, a language spoken by approximately seven million people. This performance disparity stems not from inherent model limitations but from a critical scarcity of high-quality training data. Decades of Kashmiri literature remain inaccessible to modern NLP pipelines due to their encoding in the proprietary InPage desktop publishing format. This paper introduces KS-LIT-3M, a curated corpus of 3.1 million words (16.4 million characters) specifically designed for pretraining language models on Kashmiri. The dataset is structured as a single continuous linear text stream, optimized for causal language model training where models learn to predict subsequent tokens from preceding context. The corpus was constructed through the development of a specialized InPage-to-Unicode converter, followed by rigorous preprocessing including English contamination removal, character normalization, and quality validation. Encompassing 131,607 unique words drawn from diverse genres including literary works, journalistic writing, academic texts, and religious scholarship, KS-LIT-3M addresses a fundamental resource gap for Kashmiri language technology. The dataset is released under the CC-BY-4.0 license to facilitate research in Kashmiri natural language processing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
