NumLLM：面向中文金融的数值敏感大语言模型
NumLLM: Numeric-Sensitive Large Language Model for Chinese Finance
AI总结：
本文提出面向中文金融的数值敏感大语言模型NumLLM，通过构建金融教材语料并训练两个LoRA模块分别实现领域适配与数值理解增强，在金融问答基准上取得最佳整体性能。
摘要：
近年来，许多工作通过在金融语料上从零开始预训练或微调开源大语言模型（LLM），提出了各种金融大语言模型（FinLLM）。然而，当问题中涉及数值变量时，现有 FinLLM 在理解金融文本方面表现不佳。本文提出了一种新型 LLM，称为数值敏感大语言模型（NumLLM），用于中文金融。我们首先从金融教材中构建金融语料库，这对于在微调过程中提升 LLM 的数值能力至关重要。之后，我们通过在所构建的金融语料库上进行微调，训练两个独立的低秩适配（LoRA）模块。一个模块用于将通用 LLM 适配到金融领域，另一个模块用于增强 NumLLM 理解含数值变量的金融文本的能力。最后，我们将两个 LoRA 模块合并到基础模型中，以获得用于推理的 NumLLM。在金融问答基准上的实验表明，NumLLM 能够提升基础模型的性能，并且在数值和非数值问题上均取得了相较于所有基线的最佳整体性能。
英文摘要：
Recently, many works have proposed various financial large language models (FinLLMs) by pre-training from scratch or fine-tuning open-sourced LLMs on financial corpora. However, existing FinLLMs exhibit unsatisfactory performance in understanding financial text when numeric variables are involved in questions. In this paper, we propose a novel LLM, called numeric-sensitive large language model (NumLLM), for Chinese finance. We first construct a financial corpus from financial textbooks which is essential for improving numeric capability of LLMs during fine-tuning. After that, we train two individual low-rank adaptation (LoRA) modules by fine-tuning on our constructed financial corpus. One module is for adapting general-purpose LLMs to financial domain, and the other module is for enhancing the ability of NumLLM to understand financial text with numeric variables. Lastly, we merge the two LoRA modules into the foundation model to obtain NumLLM for inference. Experiments on financial question-answering benchmark show that NumLLM can boost the performance of the foundation model and can achieve the best overall performance compared to all baselines, on both numeric and non-numeric questions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
