衡量台湾 Mandarin 语言理解能力
Measuring Taiwanese Mandarin Language Understanding
AI总结：
本文提出面向台湾 Mandarin 的 LLM 评估基准 TMLU，涵盖 37 个学科，并通过 24 个模型的实验揭示繁体中文模型性能落后于简体中文及专有多语言模型，旨在推动本地化模型发展。
摘要：
近年来，大型语言模型（LLMs）的评估在领域内引起了广泛关注。本研究聚焦于在中文语境下评估 LLMs，特别是针对在现有基准中严重代表性不足的繁体中文。我们提出 TMLU，一个专为评估 LLMs 在台湾 Mandarin 语境下的高级知识与推理能力而设计的整体评估套件。TMLU 包含 37 个学科，涵盖社会科学、STEM、人文学科、台湾特定内容及其他领域，难度范围从中学到专业级别。此外，我们为每个学科整理了类似思维链的少样本解释，以促进对复杂推理技能的评估。为建立全面的基线，我们对 24 个先进 LLM 进行了广泛的实验与分析。结果表明，中文开源权重模型相较于多语言专有模型表现较差，且针对台湾 Mandarin 优化的开源权重模型落后于简体中文对应模型。这些发现表明仍有很大的改进空间，并强调了 TMLU 促进本地化台湾 Mandarin LLM 发展的目标。我们公开发布该基准及评估脚本，以推动未来研究。
英文摘要：
The evaluation of large language models (LLMs) has drawn substantial attention in the field recently. This work focuses on evaluating LLMs in a Chinese context, specifically, for Traditional Chinese which has been largely underrepresented in existing benchmarks. We present TMLU, a holistic evaluation suit tailored for assessing the advanced knowledge and reasoning capability in LLMs, under the context of Taiwanese Mandarin. TMLU consists of an array of 37 subjects across social science, STEM, humanities, Taiwan-specific content, and others, ranging from middle school to professional levels. In addition, we curate chain-of-thought-like few-shot explanations for each subject to facilitate the evaluation of complex reasoning skills. To establish a comprehensive baseline, we conduct extensive experiments and analysis on 24 advanced LLMs. The results suggest that Chinese open-weight models demonstrate inferior performance comparing to multilingual proprietary ones, and open-weight models tailored for Taiwanese Mandarin lag behind the Simplified-Chinese counterparts. The findings indicate great headrooms for improvement, and emphasize the goal of TMLU to foster the development of localized Taiwanese-Mandarin LLMs. We release the benchmark and evaluation scripts for the community to promote future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
