arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.00664cs.CL
收藏

YOMI-Bench:评估日语大语言模型汉字读音与音韵理解的基准

YOMI-Bench: A Benchmark for Evaluating Kanji Reading and Phonological Understanding of LLMs for Japanese

Kobe University(神户大学);National Institute of Advanced Industrial Science and Technology (AIST)(国立研究开发法人产业技术综合研究所(AIST));The University of Tokyo(东京大学);RIKEN(理化学研究所);Tohoku University(东北大学)

Ryota Mibayashi, Hiroya Takamura, Hitomi Yanaka

AI总结:

提出YOMI-Bench基准,通过四个任务评估大语言模型在日语汉字读音上的表现,发现专用模型和商业模型均表现不佳。

摘要:

我们提出了YOMI-Bench,一个用于评估大语言模型(LLMs)在日语中汉字读音和音韵理解的基准。在日语中,单个汉字字符通常有多种可能的读音,使得仅从表面文本推断正确读音变得困难。由于这些语言特性,经验上已知LLMs在日语汉字读音方面表现较低。所提出的YOMI-Bench由四个专门设计用于评估日语汉字读音表现的任务组成。在使用YOMI-Bench的评估中,我们评估了一个多语言开源LLM、四个日语专用开源LLM和五个商业LLM。结果发现,即使是日语专用模型也表现较低,并且商业模型在需要考虑汉字读音的生成任务上也表现不佳。

英文摘要:

We propose YOMI-Bench, a benchmark for evaluating kanji reading and phonological understanding of large language models (LLMs) for Japanese. In Japanese, a single kanji character often has multiple possible readings, making it difficult to infer the correct reading from surface-level text alone. Due to these linguistic characteristics, it is empirically known that LLMs exhibit low performance in kanji reading for Japanese. The proposed YOMI-Bench consists of four tasks specifically designed to evaluate kanji reading performance in Japanese. In our evaluation using YOMI-Bench, we assessed one multilingual open LLM, four Japanese-specific open LLMs, and five commercial LLMs. As a result, we found that even Japanese-specific models show low performance, and that commercial models also perform poorly on generation tasks that require consideration of kanji readings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑