语言模型能否处理非公历日历?以日本和历为例
Can Language Models Handle a Non-Gregorian Calendar? The Case of the Japanese wareki
AI总结:
针对现有研究仅聚焦公历的局限,本文构建日本和历时间推理数据集,评估发现部分模型能完成日历转换,但主流模型在和历算术与知识任务中表现不佳,凸显开发适配文化特定日历理解的LMs的重要性。
摘要:
时间推理和知识是语言模型(LMs)的核心能力。尽管此前大量研究分析并改进了LMs的时间推理能力,但多数研究仅聚焦公历。然而,许多非公历系统(如日本、伊斯兰和希伯来历法)仍在广泛使用,反映了文化背景下的时间概念。目前尚未评估当前LMs能否准确处理这类非公历日历及处理效果。本文系统评估LMs对日本和历这一非公历系统的处理能力,构建了需运用和历时间知识与推理的数据集。评估开源及闭源LMs后发现,部分模型能完成日历转换,但GPT-4o、Deepseek V3乃至日语专用模型在和历算术与知识任务中表现不佳。错误分析表明,和历表达式的语料库频率及模型知识中的公历偏见可能是原因。研究结果凸显了开发更适配文化特定任务(如日历理解)的LMs的重要性。
英文摘要:
Temporal reasoning and knowledge are essential capabilities for language models (LMs). While much prior work has analyzed and improved temporal reasoning in LMs, most studies have focused solely on the Gregorian calendar. However, many non-Gregorian systems, such as the Japanese, Hijri, and Hebrew calendars, are in active use and reflect culturally grounded conceptions of time. If and how well current LMs can accurately handle such non-Gregorian calendars has not been evaluated so far. Here, we present a systematic evaluation of how well language models handle one such non-Gregorian system: the Japanese wareki. We create datasets that require temporal knowledge and reasoning in using wareki dates. Evaluating open and closed LMs, we find that some models can perform calendar conversions, but GPT-4o, Deepseek V3, and even Japanese-centric models struggle with Japanese calendar arithmetic and knowledge involving wareki dates. Error analysis suggests corpus frequency of Japanese calendar expressions and a Gregorian bias in the model's knowledge as possible explanations. Our results show the importance of developing LMs that are better equipped for culture-specific tasks such as calendar understanding.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。