大型语言模型了解民间故事吗？以日本民间故事中的妖怪为案例研究
Do Large Language Models Know Folktales? A Case Study of Yokai in Japanese Folktales
AI总结：
本文通过构建包含809道选择题的YokaiEval基准，评估31个日语及多语言LLMs对日本妖怪知识的掌握，发现日语训练模型优于英语中心模型，且基于Llama-3的日语持续预训练模型表现最佳。
摘要：
尽管大型语言模型（LLMs）已在多种语言上展现出强大的语言理解与生成能力，但其文化知识往往局限于英语社群，这可能使非英语社群的文化被边缘化。为解决这一问题，研究者已开始评估LLMs的文化意识，并探索开发具有文化意识的LLMs的方法。本研究聚焦于评估模型对民间故事这一文化传播与流转关键媒介的知识，尤其关注日本民间故事，具体考察其对妖怪（Yokai）的知识。妖怪是源自日本民间故事的超自然生物，至今仍是艺术与娱乐中的流行母题。妖怪长期作为文化表达的媒介，使其成为评估LLMs文化意识的理想对象。我们引入了YokaiEval，一个由809道多项选择题（每题四个选项）组成的基准数据集，旨在探测模型关于妖怪的知识。我们评估了31个日语及多语言LLMs在该数据集上的表现。结果表明，使用日语资源训练的模型比以英语为中心的模型取得了更高的准确率，其中经过日语持续预训练的模型表现尤为突出，尤其是基于Llama-3的模型。代码和数据集可在https://github.com/CyberAgentAILab/YokaiEval获取。
英文摘要：
Although Large Language Models (LLMs) have demonstrated strong language understanding and generation abilities across various languages, their cultural knowledge is often limited to English-speaking communities, which can marginalize the cultures of non-English communities. To address the problem, evaluation of the cultural awareness of the LLMs and the methods to develop culturally aware LLMs have been investigated. In this study, we focus on evaluating knowledge of folktales, a key medium for conveying and circulating culture. In particular, we focus on Japanese folktales, specifically on knowledge of Yokai. Yokai are supernatural creatures originating from Japanese folktales that continue to be popular motifs in art and entertainment today. Yokai have long served as a medium for cultural expression, making them an ideal subject for assessing the cultural awareness of LLMs. We introduce YokaiEval, a benchmark dataset consisting of 809 multiple-choice questions (each with four options) designed to probe knowledge about yokai. We evaluate the performance of 31 Japanese and multilingual LLMs on this dataset. The results show that models trained with Japanese language resources achieve higher accuracy than English-centric models, with those that underwent continued pretraining in Japanese, particularly those based on Llama-3, performing especially well. The code and dataset are available at https://github.com/CyberAgentA ILab/YokaiEval.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
