arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2105.09680cs.CL
收藏

KLUE:韩语语言理解评估

KLUE: Korean Language Understanding Evaluation

Upstage;KAIST(韩国科学技术院);NAVER AI Lab(NAVER AI 实验室);Seoul National University(首尔大学);Yonsei University(延世大学);Scatter Lab;KyungHee University(庆熙大学);Kakao Enterprise;Sogang University(西江大学);Hanbat National University(韩巴国立大学);Riiid AI Research(Riiid AI 研究院);New York University(纽约大学)
Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, … 展开作者

Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, Seongbo Jang, Seungwon Do, Sunkyoung Kim, Kyungtae Lim, Jongwon Lee, Kyumin Park, Jamin Shin, Seonghyun Kim, Lucy Park, Alice Oh, Jung-Woo Ha, Kyunghyun Cho

摘要:

我们介绍韩国语语言理解评估(KLUE)基准。KLUE是8个韩语自然语言理解(NLU)任务的集合,包括主题分类、语义文本相似性、自然语言推理、命名实体识别、关系抽取、依存句法分析、机器阅读理解和对话状态跟踪。我们从不同的源语料库中从头开始构建所有任务,同时尊重版权,以确保任何人都可以无限制地访问。考虑到伦理因素,我们精心设计了注释协议。除了基准任务和数据,我们还为每个任务的预训练语言模型提供了合适的评估指标和微调配方。我们还发布了预训练的语言模型(PLM)KLUE-BERT和KLUE-RoBERTa,以帮助在KLUE上重现基线模型,从而促进未来的研究。我们提出了一些有趣的意见,从初步实验中使用拟议的KLUE基准套件,已经证明了这个新的基准套件的有用性。首先,我们发现KLUE-RoBERTa-large优于其他基准,包括多语言PLM和现有的开源韩国PLM。其次,即使我们从预训练语料库中替换个人身份信息,我们也会看到性能的最小下降,这表明隐私和NLU能力并不矛盾。最后,我们发现,使用BPE标记与词素级预标记相结合是有效的任务,涉及词素级标记,检测和生成。除了加速韩语NLP研究之外,我们关于创建KLUE的全面文档将有助于将来为其他语言创建类似的资源。KLUE可在https://klue-benchmark.com上获得。

英文摘要:

We introduce Korean Language Understanding Evaluation (KLUE) benchmark. KLUE is a collection of 8 Korean natural language understanding (NLU) tasks, including Topic Classification, SemanticTextual Similarity, Natural Language Inference, Named Entity Recognition, Relation Extraction, Dependency Parsing, Machine Reading Comprehension, and Dialogue State Tracking. We build all of the tasks from scratch from diverse source corpora while respecting copyrights, to ensure accessibility for anyone without any restrictions. With ethical considerations in mind, we carefully design annotation protocols. Along with the benchmark tasks and data, we provide suitable evaluation metrics and fine-tuning recipes for pretrained language models for each task. We furthermore release the pretrained language models (PLM), KLUE-BERT and KLUE-RoBERTa, to help reproducing baseline models on KLUE and thereby facilitate future research. We make a few interesting observations from the preliminary experiments using the proposed KLUE benchmark suite, already demonstrating the usefulness of this new benchmark suite. First, we find KLUE-RoBERTa-large outperforms other baselines, including multilingual PLMs and existing open-source Korean PLMs. Second, we see minimal degradation in performance even when we replace personally identifiable information from the pretraining corpus, suggesting that privacy and NLU capability are not at odds with each other. Lastly, we find that using BPE tokenization in combination with morpheme-level pre-tokenization is effective in tasks involving morpheme-level tagging, detection and generation. In addition to accelerating Korean NLP research, our comprehensive documentation on creating KLUE will facilitate creating similar resources for other languages in the future. KLUE is available at https://klue-benchmark.com.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑