arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.04693cs.CL
收藏

Thunder-KoNUBench: 一个与语料库对齐的韩语否定理解基准

Thunder-KoNUBench: A Corpus-Aligned Benchmark for Korean Negation Understanding

Sungmok Jung, Yeonkyoung So, Joonhak Lee, Sangho Kim, Yelim Ahn, Jaejin Lee

AI总结:

本文提出Thunder-KoNUBench,通过分析韩语否定现象,评估47个LLM在否定理解上的表现,揭示模型大小和指令微调的影响,并展示微调提升韩语否定理解和上下文理解的效果。

摘要:

尽管否定对大型语言模型(LLMs)构成挑战,但针对韩语否定的评估基准却十分稀缺。我们对韩语否定进行了基于语料库的分析,证明LLM在否定任务上的表现下降。随后引入Thunder-KoNUBench,这是一个句子级否定理解基准,反映了韩语否定现象的实证分布。在Thunder-KoNUBench上评估47个LLM,分析模型大小和指令微调的影响,并进行错误分析以更好地理解模型行为。我们进一步表明,在Thunder-KoNUBench上进行微调可以提升韩语否定理解和更广泛的上下文理解能力。

英文摘要:

Although negation is known to challenge large language models (LLMs), benchmarks for evaluating negation understanding-especially in Korean-are scarce. We conduct a corpus-based analysis of Korean negation and show that LLM performance degrades under negation. We then introduce Thunder-KoNUBench, a sentence-level negation understanding benchmark that reflects the empirical distribution of Korean negation phenomena. Evaluating 47 LLMs on Thunder-KoNUBench, we analyze the effects of model size and instruction tuning, and perform error analysis to better understand model behavior. We further show that fine-tuning on Thunder-KoNUBench improves negation understanding and broader contextual comprehension in Korean.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑