发表机构
Bilibili（哔哩哔哩）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
哔哩哔哩开发Index-1.9B系列小语言模型，包括基础模型等四个模型。预训练采用特定方法，Index-1.9B-Base在标准基准测试表现出色，还进行了多项控制研究，所有模型及评估代码已公开。
AI 中文摘要
我们展示了哔哩哔哩开发的Index-1.9B系列开放小语言模型。该系列包含四个模型：Index-1.9B-Base，在2.8万亿主要中英文token上预训练的含19亿非嵌入参数的基础模型；Index-1.9B-Pure，用相同方法训练但严格过滤指令类数据的控制变体；Index-1.9B-Chat，通过监督微调与直接偏好优化从基础模型对齐；Index-1.9B-Character，通过检索增强生成进行少样本角色扮演定制来增强聊天模型。预训练采用特定学习率调度，配合Norm-Head输出层。Index-1.9B-Base在标准基准测试中取得平均64.92分，与数倍于其规模的开放模型竞争甚至超越。还报告了关于模型深度、学习率大小和调度等的控制研究，所有模型及评估代码已发布。
英文摘要
We present Index-1.9B, a series of open small language models developed at Bilibili. The series comprises four models: Index-1.9B-Base, a foundation model with 1.9 billion non-embedding parameters pre-trained on 2.8 trillion predominantly Chinese and English tokens; Index-1.9B-Pure, a control variant trained with an identical recipe but with all instruction-like data strictly filtered from the corpus; Index-1.9B-Chat, aligned from the base model with supervised fine-tuning and direct preference optimization; and Index-1.9B-Character, which augments the chat model with retrieval-augmented generation for few-shot role-playing customization. Pre-training employs a Warmup-Stable-Decay learning-rate schedule in which the concentration of curated data is raised substantially during the decay phase, together with a Norm-Head output layer that stabilizes training under large learning rates. On a suite of standard benchmarks covering examination, reasoning, mathematics, and code, Index-1.9B-Base attains an average score of 64.92, competitive with or exceeding open models of several times its size. We further report controlled studies on model depth, learning-rate magnitude and scheduling, the interaction between learning-rate decay and data quality, and the effect of including instruction data during pre-training, and we document an unexplained surge in benchmark performance midway through the constant-learning-rate phase. All models, together with evaluation code, are released at https://github.com/bilibili/Index-1.9B.
Comments16 pages, 9 figures. v3: updated author list to add Xipeng Wang
