ERNIE 5.0技术报告
ERNIE 5.0 Technical Report
AI总结:
ERNIE 5.0是首个实现万亿参数统一自回归模型的生产规模实现,支持多模态理解和生成,并采用弹性训练范式实现灵活的性能与资源权衡。
摘要:
在本报告中,我们介绍了ERNIE 5.0,这是一种原生自回归的基础模型,旨在实现跨文本、图像、视频和音频的统一多模态理解和生成。所有模态均基于一种超稀疏专家混合(MoE)架构进行从头训练,该架构采用模态无关的专家路由机制,基于统一的下一个token组预测目标。为了解决大规模部署在多样资源约束下的实际挑战,ERNIE 5.0采用了新的弹性训练范式。在单次预训练运行中,模型学习了一组具有不同深度、专家容量和路由稀疏性的子模型,从而在内存或时间受限的场景中实现性能、模型大小和推理延迟之间的灵活权衡。此外,我们系统性地解决了将强化学习扩展到统一基础模型的挑战,从而确保在超稀疏MoE架构和多样化的多模态设置下实现高效稳定的后期训练。大量实验表明,ERNIE 5.0在多种模态上实现了强大且均衡的性能。据我们所知,在公开披露的模型中,ERNIE 5.0是首个实现万亿参数统一自回归模型的生产规模实现,该模型支持多模态理解和生成。为了促进进一步研究,我们展示了统一模型中模态无关专家路由的详细可视化,并提供了弹性训练的全面实证分析,旨在为社区提供深刻见解。
英文摘要:
In this report, we introduce ERNIE 5.0, a natively autoregressive foundation model desinged for unified multimodal understanding and generation across text, image, video, and audio. All modalities are trained from scratch under a unified next-group-of-tokens prediction objective, based on an ultra-sparse mixture-of-experts (MoE) architecture with modality-agnostic expert routing. To address practical challenges in large-scale deployment under diverse resource constraints, ERNIE 5.0 adopts a novel elastic training paradigm. Within a single pre-training run, the model learns a family of sub-models with varying depths, expert capacities, and routing sparsity, enabling flexible trade-offs among performance, model size, and inference latency in memory- or time-constrained scenarios. Moreover, we systematically address the challenges of scaling reinforcement learning to unified foundation models, thereby guaranteeing efficient and stable post-training under ultra-sparse MoE architectures and diverse multimodal settings. Extensive experiments demonstrate that ERNIE 5.0 achieves strong and balanced performance across multiple modalities. To the best of our knowledge, among publicly disclosed models, ERNIE 5.0 represents the first production-scale realization of a trillion-parameter unified autoregressive model that supports both multimodal understanding and generation. To facilitate further research, we present detailed visualizations of modality-agnostic expert routing in the unified model, alongside comprehensive empirical analysis of elastic training, aiming to offer profound insights to the community.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。