Fann or Flop：面向大语言模型的多体裁、多时代阿拉伯诗歌理解基准
Fann or Flop: A Multigenre, Multiera Benchmark for Arabic Poetry Understanding in LLMs
AI总结：
提出首个多时代、多体裁的阿拉伯诗歌理解基准Fann or Flop，评估LLMs在12个历史时代、14种体裁下的语义、隐喻、韵律和文化理解，发现现有模型表现不佳，并开源资源以推动进步。
摘要：
阿拉伯诗歌是阿拉伯语言中最丰富、最具文化根基的表达形式之一，以其多层次的涵义、风格多样性和深厚的历史延续性而著称。尽管大语言模型（LLMs）已在多种语言和任务上展现出强劲性能，但其对阿拉伯诗歌的理解能力在很大程度上仍未得到探索。在这项工作中，我们推出了“Fann or Flop”，这是首个旨在评估LLMs在12个历史时代、涵盖14种核心诗歌体裁及多种格律形式（从古典结构到当代自由诗）下对阿拉伯诗歌理解能力的基准。该基准包含一个经过策展的诗歌语料库，并附有解释，用于评估语义理解、隐喻解读、韵律意识和文化背景。我们认为，诗歌理解力是检验LLM通过阿拉伯诗歌理解古典阿拉伯语能力的一个有力指标。与表面层面的任务不同，这一领域要求更深层次的解释性推理和文化敏感性。我们对最先进LLMs的评估显示，尽管在标准阿拉伯语基准上表现强劲，但大多数模型在诗歌理解方面仍存在困难。我们将“Fann or Flop”及其评估套件作为开源资源发布，以促进对阿拉伯语言模型的严格评估和进步。代码可在 https://github.com/mbzuai-oryx/FannOrFlop 获取。
英文摘要：
Arabic poetry is one of the richest and most culturally rooted forms of expression in the Arabic language, known for its layered meanings, stylistic diversity, and deep historical continuity. Although large language models (LLMs) have demonstrated strong performance across languages and tasks, their ability to understand Arabic poetry remains largely unexplored. In this work, we introduce \emph{Fann or Flop}, the first benchmark designed to assess the comprehension of Arabic poetry by LLMs in 12 historical eras, covering 14 core poetic genres and a variety of metrical forms, from classical structures to contemporary free verse. The benchmark comprises a curated corpus of poems with explanations that assess semantic understanding, metaphor interpretation, prosodic awareness, and cultural context. We argue that poetic comprehension offers a strong indicator for testing how good the LLM understands classical Arabic through Arabic poetry. Unlike surface-level tasks, this domain demands deeper interpretive reasoning and cultural sensitivity. Our evaluation of state-of-the-art LLMs shows that most models struggle with poetic understanding despite strong results on standard Arabic benchmarks. We release "Fann or Flop" along with the evaluation suite as an open-source resource to enable rigorous evaluation and advancement for Arabic language models. Code is available at: https://github.com/mbzuai-oryx/FannOrFlop.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
