阿姆哈拉语故事问答基准：阿姆哈拉语多文化故事问答基准
AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic
AI总结：
本文提出AmharicStoryQA基准，通过多文化叙事揭示现有LLMs在叙事理解上的不足，强调文化差异对语言评估的影响。
摘要：
随着对大型语言模型的多语言和文化评估基准日益重视，语言和文化常被视为同义词，性能常被用作模型对给定语言理解的代理。在本工作中，我们主张此类评估忽视了单种语言内部存在的有意义的文化差异。我们通过聚焦埃塞俄比亚不同地区的叙事来填补这一空白，并证明尽管有共享的语言特征，地区特定和领域特定的内容显著影响语言评估结果。为此，我们引入了\textbf{\textit{AmharicStoryQA}}，一个基于阿姆哈拉语地区文化多样叙事的长序列故事问答基准。利用此基准，我们揭示了现有LLMs在叙事理解上的显著差距，突显了评估结果中的明显地区差异，并展示了监督微调在不同地区和评估设置中产生的不均衡改进。我们的发现强调了需要文化导向的基准，以超越语言层面的评估，更准确地评估和改进低资源语言的叙事理解。
英文摘要：
With the growing emphasis on multilingual and cultural evaluation benchmarks for large language models, language and culture are often treated as synonymous, and performance is commonly used as a proxy for a models understanding of a given language. In this work, we argue that such evaluations overlook meaningful cultural variation that exists within a single language. We address this gap by focusing on narratives from different regions of Ethiopia and demonstrate that, despite shared linguistic characteristics, region-specific and domain-specific content substantially influences language evaluation outcomes. To this end, we introduce \textbf{\textit{AmharicStoryQA}}, a long-sequence story question answering benchmark grounded in culturally diverse narratives from Amharic-speaking regions. Using this benchmark, we reveal a significant narrative understanding gap in existing LLMs, highlight pronounced regional differences in evaluation results, and show that supervised fine-tuning yields uneven improvements across regions and evaluation settings. Our findings emphasize the need for culturally grounded benchmarks that go beyond language-level evaluation to more accurately assess and improve narrative understanding in low-resource languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
