THaLLE-ThaiLLM:面向金融和泰语的领域专用小型LLM——技术报告
THaLLE-ThaiLLM: Domain-Specialized Small LLMs for Finance and Thai -- Technical Report
AI总结:
THaLLE-ThaiLLM通过模型合并技术,开发了面向金融和泰语的专用小型LLM,提升了泰语通用能力和金融领域性能。
摘要:
大型语言模型(LLMs)在各种领域中展现了显著的潜力,特别是在银行业和金融领域,它们可以自动化复杂任务并大规模提升决策质量。由于隐私、安全和监管方面的担忧,组织通常更倾向于在本地部署LLMs。ThaiLLM计划旨在增强开放LLMs中的泰语能力,使泰国产业能够利用先进的语言模型。然而,组织常常面临在部署多个专用模型与训练单个多功能模型的高昂成本之间的权衡。为了解决这一问题,我们探索了模型合并作为一种资源高效的替代方法,用于开发高性能的多功能LLMs。我们展示了两个关键实验的结果:首先,将Qwen-8B与ThaiLLM-8B合并,展示了ThaiLLM-8B如何提升泰语通用能力,证明了在M3和M6 O-NET考试中相比通用指令遵循的Qwen-8B有所提升。其次,我们合并Qwen-8B与ThaiLLM-8B和THaLLE-CFA-8B。这种组合在通用和金融领域均表现出进一步的性能提升,通过在M3和M6 O-NET、Flare-CFA和Thai-IC基准测试中均有所提升。本报告展示了模型合并在高效创建多功能LLMs方面的可行性。
英文摘要:
Large Language Models (LLMs) have demonstrated significant potential across various domains, particularly in banking and finance, where they can automate complex tasks and enhance decision-making at scale. Due to privacy, security, and regulatory concerns, organizations often prefer on-premise deployment of LLMs. The ThaiLLM initiative aims to enhance Thai language capabilities in open-LLMs, enabling Thai industry to leverage advanced language models. However, organizations often face a trade-off between deploying multiple specialized models versus the prohibitive expense of training a single multi-capability model. To address this, we explore model merging as a resource-efficient alternative for developing high-performance, multi-capability LLMs. We present results from two key experiments: first, merging Qwen-8B with ThaiLLM-8B demonstrates how ThaiLLM-8B enhances Thai general capabilities, showing an uplift of M3 and M6 O-NET exams over the general instruction-following Qwen-8B. Second, we merge Qwen-8B with both ThaiLLM-8B and THaLLE-CFA-8B. This combination results in further improvements in performance across both general and financial domains, by demonstrating an uplift in both M3 and M6 O-NET, Flare-CFA, and Thai-IC benchmarks. The report showcases the viability of model merging for efficiently creating multi-capability LLMs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。