arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.19331cs.CLcs.AI
收藏

LLM-jp-4 技术报告

LLM-jp-4 Technical Report

LLM-jp, :, Chaoran Liu, Daisuke Kawahara, Hideyuki Tachibana, Hirokazu Kiyomaru, Jun Suzuki, Koshi Eguchi, Kouta Nakayama, Qianying Liu, Rio Yokota, Ryo Takahas… 展开作者

LLM-jp, :, Chaoran Liu, Daisuke Kawahara, Hideyuki Tachibana, Hirokazu Kiyomaru, Jun Suzuki, Koshi Eguchi, Kouta Nakayama, Qianying Liu, Rio Yokota, Ryo Takahashi, Sadao Kurohashi, Sosuke Hosokawa, Taishi Nakamura, Takashi Kodama, Yun-Ang Wu, Yusuke Miyao, Yusuke Oda

AI总结:

该研究推出日本主权大语言模型系列 LLM-jp-4,含 8B 稠密与 32B-A3B 混合专家模型,经 12 万亿 token 训练,性能优于前代且与同规模开放模型相当,为主权 AI 发展提供参考。

摘要:

本文介绍了 LLM-jp-4,这是一款日本主权大语言模型(LLM)系列,开发时着重关注透明性、可复现性和实际可用性。LLM-jp-4 包含一个 8B 稠密模型和一个 32B-A3B 混合专家模型,两者均从零开始在约 12 万亿个 token 上训练而成。为支持独立验证与复现,我们制定了训练数据、外部模型及软件的明确选择标准,并公开了开发过程。评估结果显示,其较前代 LLM-jp 模型有显著提升,且与同等规模的开放权重模型性能相当。通过记录 LLM-jp-4 的开发原则、训练流程及实证发现,我们旨在为未来主权 AI 的发展提供参考。

英文摘要:

This paper presents LLM-jp-4, a family of Japanese sovereign large language models (LLMs) developed with an emphasis on transparency, reproducibility, and practical usability. LLM-jp-4 comprises an 8B dense model and a 32B-A3B mixture-of-experts model, both trained from scratch on approximately 12 trillion tokens. To support independent verification and reproduction, we establish explicit criteria for selecting training data, external models, and software, and make the development process publicly available. Evaluation results demonstrate substantial improvements over previous LLM-jp models and competitive performance against open-weight models of similar scale. By documenting the development principles, training procedures, and empirical findings of LLM-jp-4, we aim to provide a reference for future sovereign AI development.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑