ChuXin:1.6B 技术报告
ChuXin: 1.6B Technical Report
AI总结:
本文介绍完全开源的16亿参数语言模型 ChuXin,公开训练数据、流程与评估代码,并通过持续预训练将上下文扩展至100万 token。
摘要:
在本报告中,我们介绍了 ChuXin,这是一个完全开源、参数量为16亿的语言模型。与大多数仅开源模型权重和架构的工作不同,我们公开了训练模型所需的一切内容,包括训练数据、训练流程和评估代码。我们的目标是赋能并壮大开放研究社区,促进透明度,并推动语言建模领域的新一轮创新。此外,我们通过轻量级持续预训练将上下文长度扩展至100万 token,并展示了强大的“大海捞针”检索性能。两个模型的权重均可在 Hugging Face 上下载和使用。
英文摘要:
In this report, we present ChuXin, an entirely open-source language model with a size of 1.6 billion parameters. Unlike the majority of works that only open-sourced the model weights and architecture, we have made everything needed to train a model available, including the training data, the training process, and the evaluation code. Our goal is to empower and strengthen the open research community, fostering transparency and enabling a new wave of innovation in the field of language modeling. Furthermore, we extend the context length to 1M tokens through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. The weights for both models are available at Hugging Face to download and use.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。