大语言模型(LLMs)的快速发展凸显了构建稳健评估框架以衡量其推理、知识、常识等核心能力的必要性,由此催生了H6基准等部分广泛使用的基准套件。但这些基准套件主要为英语构建,在LLM开发中代表性不足的语言(如泰语)缺乏此类资源。另一方面,泰语LLM的开发还应包括提升文化理解能力与核心能力。为应对泰语LLM研究中的这一双重挑战,我们提出两大核心基准:Thai-H6与泰语文化与语言智能基准(Thai Cultural and Linguistic Intelligence Benchmark, ThaiCLI)。通过对多款具备多语言能力的LLM进行全面评估,我们对所提基准及其对泰语LLM开发的助力作用展开了综合分析。此外,我们将公开数据集与评估代码,以推动泰语LLM的进一步研发。
英文摘要:
The rapid advancement of large language models (LLMs) has highlighted the need for robust evaluation frameworks that assess their core capabilities, such as reasoning, knowledge, and commonsense, leading to the inception of certain widely-used benchmark suites such as the H6 benchmark. However, these benchmark suites are primarily built for the English language, and there exists a lack thereof for under-represented languages, in terms of LLM development, such as Thai. On the other hand, developing LLMs for Thai should also include enhancing the cultural understanding as well as core capabilities. To address these dual challenge in Thai LLM research, we propose two key benchmarks: Thai-H6 and Thai Cultural and Linguistic Intelligence Benchmark (ThaiCLI). Through a thorough evaluation of various LLMs with multi-lingual capabilities, we provide a comprehensive analysis of the proposed benchmarks and how they contribute to Thai LLM development. Furthermore, we will make both the datasets and evaluation code publicly available to encourage further research and development for Thai LLMs.
DIWALI: Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context
Natural Language and Information Processing Lab (NLIP) Indian Institute of Technology Hyderabad(自然语言与信息处理实验室(NLIP)印度理工学院海得拉巴)
Meaning Is Not A Metric: Using LLMs to make cultural context legible at scale
The Alan Turing Institute(艾伦·图灵研究所);University of Edinburgh(爱丁堡大学);University of Colorado Boulder(科罗拉多大学丹佛分校);University of Chicago(芝加哥大学);University of Exeter(埃克塞特大学);University of Southampton(南安普顿大学)
WorldView-Bench: A Benchmark for Evaluating Global Cultural Perspectives in Large Language Models
Information Technology University Department of Computer Science(信息科技大学计算机科学系);Zayed University College of Interdisciplinary Studies(扎耶德大学跨学科研究学院);Computer Science and Engineering Department(计算机科学与工程系)
NormAd: A Framework for Measuring the Cultural Adaptability of Large Language Models
Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所);Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院)
The Ghost in the Machine has an American accent: value conflict in GPT-3
The University of Sydney(悉尼大学);Sorbonne Université(索邦大学);European University Institute(欧洲大学学院);Ruhr Universität Bochum(波鸿鲁尔大学);University of Sassari(萨萨里大学);Vytautas Magnus University(维陶塔斯·马格努斯大学);Delft University of Technology(代尔夫特理工大学)
C-VARC: A Large-Scale Chinese Value Rule Corpus for Value Alignment of Large Language Models
BrainCog Lab, Institute of Automation, Chinese Academy of Sciences, Beijing, China(中国科学院自动化研究所脑认知实验室,北京,中国);Beijing Key Laboratory of Safe AI and Superalignment, Beijing, China(北京安全人工智能与超对齐关键实验室,北京,中国);Beijing Institute of AI Safety and Governance, Beijing, China(北京人工智能安全与治理研究所,北京,中国);The School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China(中国科学院大学人工智能学院,北京,中国);Long-term AI, Beijing, China(长期人工智能,北京,中国)
Schema-Guided Culture-Aware Complex Event Simulation with Multi-Agent Role-Play
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);The Private Sector Humanitarian Alliance(私营部门人道主义联盟);DEVCOM Army Research Laboratory(DEVCOM陆军研究实验室)
When Meaning Travels: A Granular Lens on Hybrid-MoE's Role in Idiomatic Understanding for Language Models
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India(印度理工学院帕纳瓦分校计算机科学与工程系);School of Information Technology, King Mongkut’s Institute of Technology Ladkrabang, Thailand(泰国拉差班国王理工大学信息科技学院)
Lutma: a Frame-Making Tool for Collaborative FrameNet Development
Federal University of Juiz de Fora(茹伊斯迪福拉联邦大学);Brazilian National Council for Scientific and Technological Development – CNPq(巴西国家科学技术发展委员会(CNPq));Federal University of Minas Gerais(米纳斯吉拉斯联邦大学)
Vashantor: A Large-scale Multilingual Benchmark Dataset for Automated Translation of Bangla Regional Dialects to Bangla Language
Department of Computer Science and Engineering, Ahsanullah University of Science and Technology(阿斯南大学科学与技术学院);Environmental Sciences, Khulna University(库尔纳大学环境科学系);Department of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院)