Bielik 11B v2 技术报告
Bielik 11B v2 Technical Report
AI总结:
本文介绍面向波兰语的 Bielik 11B v2 模型,通过深度上扩展将 Mistral 架构扩至 11B 参数,并引入加权指令交叉熵损失与自适应学习率,以高效参数规模超越更大模型,推动低资源语言建模发展。
摘要:
我们提出了 Bielik 11B v2,一个针对波兰语文本处理优化的最先进语言模型。该模型基于 Mistral 7B v0.2 架构构建,并通过深度上扩展(depth up-scaling)将参数规模扩展至 11B,在波兰语基准测试中展现出卓越性能,同时保持强大的跨语言能力。我们引入了两项关键技术创新:加权指令交叉熵损失(Weighted Instruction Cross-Entropy Loss),通过为训练样本分配基于质量的权重来优化对不同指令类型的学习;以及自适应学习率(Adaptive Learning Rate),根据上下文长度动态调整学习率。在多个基准上的综合评估表明,Bielik 11B v2 超越了许多更大的模型,包括参数量为其 2 至 6 倍的模型,并在从语言理解到复杂推理的任务上显著优于其他专用波兰语模型。该模型的参数效率及广泛的量化选项使其能够部署于各种硬件配置,推动了波兰语人工智能能力的发展,并为低资源语言的高效语言建模设立了新的基准。
英文摘要:
We present Bielik 11B v2, a state-of-the-art language model optimized for Polish text processing. Built on the Mistral 7B v0.2 architecture and scaled to 11B parameters using depth up-scaling, this model demonstrates exceptional performance across Polish language benchmarks while maintaining strong cross-lingual capabilities. We introduce two key technical innovations: Weighted Instruction Cross-Entropy Loss, which optimizes learning across diverse instruction types by assigning quality-based weights to training examples, and Adaptive Learning Rate, which dynamically adjusts based on context length. Comprehensive evaluation across multiple benchmarks demonstrates that Bielik 11B v2 outperforms many larger models, including those with 2-6 times more parameters, and significantly surpasses other specialized Polish language models on tasks ranging from linguistic understanding to complex reasoning. The model's parameter efficiency and extensive quantization options enable deployment across various hardware configurations, advancing Polish language AI capabilities and establishing new benchmarks for resource-efficient language modeling in less-represented languages.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。