arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2204.04541cs.CL
收藏

KOBEST:韩国重大任务的平衡评估

KOBEST: Korean Balanced Evaluation of Significant Tasks

SK Telecom(SK电讯);University of Oxford(牛津大学)

Dohyeong Kim, Myeongjun Jang, Deuk Sin Kwon, Eric Davis

摘要:

一个良好的基准在推动自然语言处理(NLP)领域的进步方面发挥着关键作用,因为它允许对不同模型进行客观和精确的评估。随着现代语言模型(LM)变得越来越精细和复杂,已经提出了需要语言知识和推理的更困难的基准。然而,这些基准测试中的大多数只支持英语,并且需要为其他低资源语言构建基准测试。为此,我们提出了一个新的基准命名为韩国的平衡评估的重要任务(KoBEST),其中包括五个韩语下游任务。专业的韩语语言学家设计了需要高级韩语语言知识的任务。此外,我们的数据完全由人类注释,并经过彻底审查,以保证高数据质量。我们还提供基准模型和人员绩效结果。我们的数据集可以在Huggingface上找到。

英文摘要:

A well-formulated benchmark plays a critical role in spurring advancements in the natural language processing (NLP) field, as it allows objective and precise evaluation of diverse models. As modern language models (LMs) have become more elaborate and sophisticated, more difficult benchmarks that require linguistic knowledge and reasoning have been proposed. However, most of these benchmarks only support English, and great effort is necessary to construct benchmarks for other low resource languages. To this end, we propose a new benchmark named Korean balanced evaluation of significant tasks (KoBEST), which consists of five Korean-language downstream tasks. Professional Korean linguists designed the tasks that require advanced Korean linguistic knowledge. Moreover, our data is purely annotated by humans and thoroughly reviewed to guarantee high data quality. We also provide baseline models and human performance results. Our dataset is available on the Huggingface.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑