TinyHelen 的第一课程：在更简单的语言环境中训练和评估微型语言模型
TinyHelen's First Curriculum: Training and Evaluating Tiny Language Models in a Simpler Language Environment
AI总结：
本文提出通过构建低噪声、低词汇量且保留体裁特征的简化语言环境及精简数据集，提升微型语言模型的训练与评估效率，并验证其在指令遵循等任务上优于原始数据训练效果。
摘要：
由于大规模数据集和模型，训练语言模型（LMs）及其应用智能体的成本日益高昂，使得测试失败难以承受。简化的语言环境可作为原始的训练和测试场所，保留基本的常识和沟通技能，但以更易消化的形式呈现，从而可能提升语言模型的学习效率，并因此减少有效训练和评估所需的模型规模和数据量。在这些简化的语言环境中，适用于小模型、数据集和智能体的可行策略，可能可以迁移到复杂语言环境中的更大模型、数据集和智能体。为了创建这样的环境，我们关注两个方面：i）最小化语言数据集的噪声和复杂性，以及 ii）保留基本的文本分布特征。与以往方法不同，我们提出了一种精炼文本数据的流程，通过消除噪声、最小化词汇量并保持特定体裁模式（例如书籍、对话、代码等）。利用大型语言模型实现该流程，我们创建了一套更精简的语言模型训练和评估数据集：71M Leaner-Pretrain、7M Leaner-Instruct、用于评估语言能力的 Leaner-Glue，以及用于测试指令遵循能力的 Leaner-Eval。我们的实验表明，精简预训练提升了语言模型的学习效率。在这些数据集上训练的微型语言模型，在不同语言粒度级别的指令遵循方面，优于在原始数据集上训练的模型。此外，Leaner-Pretrain 数据集与常规大型语言模型训练集的对齐，使得能够以资源优化的方式分析学习目标、模型架构和训练技术如何影响语言建模和下游任务的性能。我们的代码和数据集可在 https://github.com/EmpathYang/TinyHelen.git 获取。
英文摘要：
Training language models (LMs) and their application agents is increasingly costly due to large datasets and models, making test failures difficult to bear. Simplified language environments serve as primordial training and testing grounds, retaining essential commonsense and communication skills but in a more digestible form, potentially enhancing the learning efficiency of LMs, and thus reducing the required model size and data volume for effective training and evaluation. In these simplified language environments, workable strategies for small models, datasets, and agents may be adaptable to larger models, datasets, and agents in complex language environments. To create such environments, we focus on two aspects: i) minimizing language dataset noise and complexity, and ii) preserving the essential text distribution characteristics. Unlike previous methods, we propose a pipeline to refine text data by eliminating noise, minimizing vocabulary, and maintaining genre-specific patterns (e.g., for books, conversation, code, etc.). Implementing this pipeline with large LMs, we have created a leaner suite of LM training and evaluation datasets: 71M Leaner-Pretrain, 7M Leaner-Instruct, Leaner-Glue for assessing linguistic proficiency, and Leaner-Eval for testing instruction-following ability. Our experiments show that leaner pre-training boosts LM learning efficiency. Tiny LMs trained on these datasets outperform those trained on original datasets in instruction-following across different language granularity levels. Moreover, the Leaner-Pretrain dataset's alignment with conventional large LM training sets enables resource-optimized analysis of how learning objectives, model architectures, and training techniques impact performance on language modeling and downstream tasks. Our code and datasets are available at https://github.com/EmpathYang/TinyHelen.git.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
