破解密码:针对印度尼西亚现实世界专业考试的多领域大语言模型评估
Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesia
AI总结:
针对现有大语言模型知识评估难以覆盖现实职业需求的问题,研究团队构建了覆盖印尼6大职业领域、含8834道题的IndoCareer数据集,评估27个大模型后发现其在本土语境强的领域表现欠佳,且打乱选项会导致金融保险领域评估不稳定。
摘要:
尽管大语言模型的知识评估主要聚焦于数学、物理等学术学科,但这类评估往往无法体现现实世界职业的实际需求。本文提出IndoCareer(印度尼西亚职业评估数据集),该数据集包含8834道选择题,旨在评估模型在各领域职业与专业认证考试中的表现。IndoCareer以印度尼西亚为研究背景,提供了丰富的本土语境,覆盖六大核心领域:(1)医疗保健、(2)保险与金融、(3)创意与设计、(4)旅游与酒店、(5)教育与培训、(6)法律。我们对27个大语言模型开展了全面评估,结果显示这些模型在保险与金融等本土语境较强的领域中表现尤为吃力。此外,在使用完整数据集时,打乱答案选项通常能让各模型的评估结果保持一致,但会在保险与金融领域中引发评估结果的不稳定性。
英文摘要:
While knowledge evaluation in large language models has predominantly focused on academic subjects like math and physics, these assessments often fail to capture the practical demands of real-world professions. In this paper, we introduce IndoCareer, a dataset comprising 8,834 multiple-choice questions designed to evaluate performance in vocational and professional certification exams across various fields. With a focus on Indonesia, IndoCareer provides rich local contexts, spanning six key sectors: (1) healthcare, (2) insurance and finance, (3) creative and design, (4) tourism and hospitality, (5) education and training, and (6) law. Our comprehensive evaluation of 27 large language models shows that these models struggle particularly in fields with strong local contexts, such as insurance and finance. Additionally, while using the entire dataset, shuffling answer options generally maintains consistent evaluation results across models, but it introduces instability specifically in the insurance and finance sectors.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。