arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节(Veterans Day),12 日无更新
arXiv 2303.04715cs.CLcs.AI
收藏

扩展BLOOM预训练以提升繁体中文支持:模型、方法与结果

Extending the Pre-Training of BLOOM for Improved Support of Traditional Chinese: Models, Methods and Results

MediaTek Research(联发科研究院);Academia Sinica(中央研究院)

Philipp Ennen, Po-Chun Hsu, Chan-Jan Hsu, Chang-Le Liu, Yen-Chen Wu, Yin-Hsiang Liao, Chin-Tung Lin, Da-Shan Shiu, Wei-Yun Ma

AI总结:

本文提出BLOOM-zh,通过额外74亿token的预训练扩展BLOOM,增强繁体中文支持,在多数繁体中文基准上优于原模型且保持英文能力。

摘要:

本文介绍了多语言语言模型BLOOM-zh,该模型增强了对繁体中文的支持。BLOOM-zh源于BigScience在2022年发布的开源BLOOM模型。基于已发布的模型,我们额外使用74亿个繁体中文和英文的token扩展了BLOOM的预训练,这些token涵盖了新闻文章、书籍、百科全书、教育材料以及口语等多种领域。为了展示BLOOM-zh的特性,我们使用现有和新创建的基准场景来评估其性能。BLOOM-zh在大多数繁体中文基准上优于其前身,同时保持了其英文能力。我们将所有模型发布给研究社区。

英文摘要:

In this paper we present the multilingual language model BLOOM-zh that features enhanced support for Traditional Chinese. BLOOM-zh has its origins in the open-source BLOOM models presented by BigScience in 2022. Starting from released models, we extended the pre-training of BLOOM by additional 7.4 billion tokens in Traditional Chinese and English covering a variety of domains such as news articles, books, encyclopedias, educational materials as well as spoken language. In order to show the properties of BLOOM-zh, both existing and newly created benchmark scenarios are used for evaluating the performance. BLOOM-zh outperforms its predecessor on most Traditional Chinese benchmarks while maintaining its English capability. We release all our models to the research community.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑