Upstage;KAIST(韩国科学技术院);NAVER AI Lab(NAVER AI 实验室);Seoul National University(首尔大学);Yonsei University(延世大学);Scatter Lab;KyungHee University(庆熙大学);Kakao Enterprise;Sogang University(西江大学);Hanbat National University(韩巴国立大学);Riiid AI Research(Riiid AI 研究院);New York University(纽约大学)
Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, … 展开作者
Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, Seongbo Jang, Seungwon Do, Sunkyoung Kim, Kyungtae Lim, Jongwon Lee, Kyumin Park, Jamin Shin, Seonghyun Kim, Lucy Park, Alice Oh, Jung-Woo Ha, Kyunghyun Cho
We introduce Korean Language Understanding Evaluation (KLUE) benchmark. KLUE is a collection of 8 Korean natural language understanding (NLU) tasks, including Topic Classification, SemanticTextual Similarity, Natural Language Inference, Named Entity Recognition, Relation Extraction, Dependency Parsing, Machine Reading Comprehension, and Dialogue State Tracking. We build all of the tasks from scratch from diverse source corpora while respecting copyrights, to ensure accessibility for anyone without any restrictions. With ethical considerations in mind, we carefully design annotation protocols. Along with the benchmark tasks and data, we provide suitable evaluation metrics and fine-tuning recipes for pretrained language models for each task. We furthermore release the pretrained language models (PLM), KLUE-BERT and KLUE-RoBERTa, to help reproducing baseline models on KLUE and thereby facilitate future research. We make a few interesting observations from the preliminary experiments using the proposed KLUE benchmark suite, already demonstrating the usefulness of this new benchmark suite. First, we find KLUE-RoBERTa-large outperforms other baselines, including multilingual PLMs and existing open-source Korean PLMs. Second, we see minimal degradation in performance even when we replace personally identifiable information from the pretraining corpus, suggesting that privacy and NLU capability are not at odds with each other. Lastly, we find that using BPE tokenization in combination with morpheme-level pre-tokenization is effective in tasks involving morpheme-level tagging, detection and generation. In addition to accelerating Korean NLP research, our comprehensive documentation on creating KLUE will facilitate creating similar resources for other languages in the future. KLUE is available at https://klue-benchmark.com.
KoreALBERT: Pretraining a Lite BERT Model for Korean Language Understanding
Samsung SDS(三星SDS)
Lite BERT(ALBERT)已经被引入以扩展用于自然语言的深度双向表示学习。由于缺乏针对韩语的预训练的ALBERT模型,最佳实践是多语言模型或诉诸于任何其他基于BERT的模型。在本文中,我们开发和预训练KoreALBERT,一个单语ALBERT模型,专门为韩国语言理解。我们引入了一个新的训练目标,即词序预测(WOP),并与现有的MLM和SOP标准一起使用相同的架构和模型参数。尽管模型参数明显较少(因此,训练更快),但我们预先训练的KoreAL BERT在6个不同的NLU任务上优于其BERT对应物。与Lan等人的英语经验结果一致,KoreALBERT似乎可以提高涉及韩语多句编码的下游任务性能。预先训练的KoreALBERT是公开的,以鼓励韩国NLP的研究和应用开发。
KoNeoBench: A Curated Evaluation Dataset for LLM Understanding of Korean Neologisms
Kyungpook National University(庆北国立大学);Daegu Gyeongbuk Institute of Science and Technology (DGIST)(大邱庆北科学技术院);Texas A&M University(德克萨斯A&M大学);Yonsei University(延世大学)
KFinEval-Pilot: A Comprehensive Benchmark Suite for Korean Financial Language Understanding
Korea Financial Telecommunications and Clearings Institute(韩国金融电信与清算院);Teddysum Inc.(泰迪萨姆公司);SELECTSTAR Inc.(SELECTSTAR公司);Konyang University(建阳大学);Seoultech(首尔科技大学);KAIST(韩国科学技术院)
KOMBO: Korean Character Representations Based on the Combination Rules of Subcharacters
Department of Artificial Intelligence, Korea University, Seoul, South Korea(人工智能系,韩国大学,首尔,韩国);Department of Computer Science and Engineering, Korea University, Seoul, South Korea(计算机科学与工程系,韩国大学,首尔,韩国)
SCRIPT: A Subcharacter Compositional Representation Injection Module for Korean Pre-Trained Language Models
Department of Artificial Intelligence, Korea University, Seoul, South Korea(人工智能系,韩国大学,首尔,韩国);Department of Computer Science and Engineering, Korea University, Seoul, South Korea(计算机科学与工程系,韩国大学,首尔,韩国)
EXECUTE: A Multilingual Benchmark for LLM Token Understanding
School of Computation, Information and Technology, TU Munich(技术学院,慕尼黑技术大学);Center for Information and Language Processing, LMU Munich(信息与语言处理中心,慕尼黑大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Munich Data Science Institute(慕尼黑数据科学研究所)