arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.08620cs.CL
收藏

JAI-1:一个以泰语为中心的大型语言模型

JAI-1: A Thai-Centric Large Language Model

Jasmine Technology Solution, Thailand;Department of Linguistics, Chulalongkorn University, Thailand(语言学系,朱拉隆功大学)

Attapol T. Rutherford, Jullajak Karnjanaekarin, Narongkorn Panitsrisit, Pontakorn Trakuekul, Sumana Sumanakul, Natchanon Pollertlam

AI总结:

JAI-1通过从较小英语模型扩展参数空间并整合泰语知识,在泰语基准上超越Typhoon2-70B,实现了保留通用智能与泰语专精的平衡。

摘要:

本技术报告介绍了JAI-1,一个拥有750亿参数的以泰语为中心的语言模型。近期泰语模型主要依赖现有的开源模型,在不进行结构修改的情况下应用额外训练以专精于泰语。然而,这种方法在注入泰语特定信息时,可能会侵蚀模型参数空间中已有的知识,因为针对通用任务优化的参数可能与新的语言需求相冲突。相比之下,JAI-1采用了升级扩展策略:从一个较小的高性能英语开源大语言模型出发,我们扩展了其参数空间,并利用新分配的能力系统地整合泰语知识。这种方法不仅保留了原始模型的通用智能,还建立了区别于其他开源模型的独特架构,使得未来的可扩展增强成为可能。在预训练期间,JAI-1接触了1.5万亿个token,其中包括超过3000亿个泰语token。随后是后训练阶段——监督微调和对齐调优——使用了超过60万个基于指令的示例。最终模型在泰语中心基准测试(IFEval-TH、MT-Bench-TH和JAI-Hall-Bench)上表现出优于Typhoon2-70B的性能,验证了其升级扩展和知识整合框架的有效性。

英文摘要:

This technical report introduces JAI-1, a Thai-centric language model with 75B parameters. Recent Thai models have primarily relied on existing open-source models, applying additional training without structural modifications to specialize in Thai. However, this approach risks eroding pre-existing knowledge in the model's parameter space during the injection of Thai-specific information, as optimized parameters for general tasks may conflict with new linguistic requirements. In contrast, JAI-1 adopts an upscaling strategy: starting from a smaller, high-performing English open-source LLM, we expanded its parameter space and utilized the newly allocated capacity to systematically integrate Thai-language knowledge. This methodology not only preserves the original model's general intelligence but also establishes a unique architecture distinct from other open-source models, enabling scalable future enhancements. During pre-training, JAI-1 was exposed to 1.5T tokens, including over 300B Thai language tokens. This was followed by post-training stages -- supervised fine-tuning and alignment tuning -- using more than 600K instruction-based examples. The final model demonstrated superior performance compared to Typhoon2-70B on Thai-centric benchmarks (IFEval-TH, MT-Bench-TH, and JAI-Hall-Bench), validating the efficacy of its upscaling and knowledge-integration framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑