arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.08262cs.CLcs.AI
收藏

领域特定大语言模型的预训练与更新:日本商业领域案例研究

Pretraining and Updates of Domain-Specific LLM: A Case Study in the Japanese Business Domain

Stockmark(斯托克马克公司);National Institute of Advanced Industrial Science and Technology(国立先进工业科学技术研究院)

Kosuke Takahashi, Takahiro Omi, Kosuke Arima, Tatsuya Ishigaki

AI总结:

本文针对日本商业领域训练并更新领域特定LLM,实验表明预训练模型提升问答准确性且不损失通用知识,更新时需混合新旧文本,并公开模型与基准。

摘要:

各种语言的大语言模型(LLM)的开发一直在推进,但非英语语言与领域特定情境的结合仍未被充分探索。本文介绍了我们训练和评估一个日本商业领域特定LLM的研究结果,该模型旨在更好地理解商业相关文档,如时事新闻、技术报告和专利。此外,该领域的LLM需要定期更新以纳入最新知识。因此,我们还报告了使用最新文章数据对该LLM进行更新的首次实验和评估结果,这是一个先前研究未涉及的重要问题设置。在我们针对目标领域问答任务新创建的基准数据集上的实验中,我们发现:(1)我们的预训练模型在不损失通用知识的情况下提高了问答准确性;(2)在更新训练数据中,最新文本与较旧文本的适当混合是必要的。我们的预训练模型和商业领域基准已公开提供,以支持进一步研究。

英文摘要:

The development of Large Language Models (LLMs) in various languages has been advancing, but the combination of non-English languages with domain-specific contexts remains underexplored. This paper presents our findings from training and evaluating a Japanese business domain-specific LLM designed to better understand business-related documents, such as the news on current affairs, technical reports, and patents. Additionally, LLMs in this domain require regular updates to incorporate the most recent knowledge. Therefore, we also report our findings from the first experiments and evaluations involving updates to this LLM using the latest article data, which is an important problem setting that has not been addressed in previous research. From our experiments on a newly created benchmark dataset for question answering in the target domain, we found that (1) our pretrained model improves QA accuracy without losing general knowledge, and (2) a proper mixture of the latest and older texts in the training data for the update is necessary. Our pretrained model and business domain benchmark are publicly available to support further studies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑