arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.04795cs.CLcs.AI
收藏

Representing the Under-Represented: 面向泰语大语言模型开发的文化与核心能力基准

Representing the Under-Represented: Cultural and Core Capability Benchmarks for Developing Thai Large Language Models

Upstage AI(上阶人工智能);Chulalongkorn University(朱拉隆功大学)

Dahyun Kim, Sukyung Lee, Yungi Kim, Attapol Rutherford, Chanjun Park

AI总结:

针对泰语大语言模型缺乏适配的核心能力与文化评估基准的问题,提出Thai-H6和ThaiCLI两大基准,经多语言LLM评估验证其价值,并将开源相关资源以助力泰语LLM发展。

摘要:

大语言模型(LLMs)的快速发展凸显了构建稳健评估框架以衡量其推理、知识、常识等核心能力的必要性,由此催生了H6基准等部分广泛使用的基准套件。但这些基准套件主要为英语构建,在LLM开发中代表性不足的语言(如泰语)缺乏此类资源。另一方面,泰语LLM的开发还应包括提升文化理解能力与核心能力。为应对泰语LLM研究中的这一双重挑战,我们提出两大核心基准:Thai-H6与泰语文化与语言智能基准(Thai Cultural and Linguistic Intelligence Benchmark, ThaiCLI)。通过对多款具备多语言能力的LLM进行全面评估,我们对所提基准及其对泰语LLM开发的助力作用展开了综合分析。此外,我们将公开数据集与评估代码,以推动泰语LLM的进一步研发。

英文摘要:

The rapid advancement of large language models (LLMs) has highlighted the need for robust evaluation frameworks that assess their core capabilities, such as reasoning, knowledge, and commonsense, leading to the inception of certain widely-used benchmark suites such as the H6 benchmark. However, these benchmark suites are primarily built for the English language, and there exists a lack thereof for under-represented languages, in terms of LLM development, such as Thai. On the other hand, developing LLMs for Thai should also include enhancing the cultural understanding as well as core capabilities. To address these dual challenge in Thai LLM research, we propose two key benchmarks: Thai-H6 and Thai Cultural and Linguistic Intelligence Benchmark (ThaiCLI). Through a thorough evaluation of various LLMs with multi-lingual capabilities, we provide a comprehensive analysis of the proposed benchmarks and how they contribute to Thai LLM development. Furthermore, we will make both the datasets and evaluation code publicly available to encourage further research and development for Thai LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑