GhazalBench: 评估大语言模型对波斯抒情诗的理解与规范表层形式访问
GhazalBench: Canonical Verse Access in LLMs across Persian Ghazals and Shakespearean Sonnets
AI总结:
提出GhazalBench基准,评估LLM在波斯抒情诗中的诗意理解与规范表层形式访问能力,发现模型普遍能理解诗意但难以生成精确诗句,而识别任务缩小差距,英语表现更好,表明训练数据差异是关键。
摘要:
波斯诗歌在伊朗文化实践中扮演着活跃角色,哈菲兹等经典诗人的诗句常被引用、释义或根据部分线索补全。支持此类交互要求语言模型不仅理解诗意,还要掌握文化规范的表层形式。我们提出GhazalBench,一个评估大语言模型(LLM)在基于使用条件下与波斯抒情诗交互的基准。与先前主要将记忆视为缺陷的研究不同,GhazalBench考察在文化基础交互中精确表层形式访问功能重要的场景。该基准评估两种互补能力:诗到散文的理解,以及在变化语义和词汇线索下的规范表层形式访问。在多个专有和开源多语言LLM中,我们观察到一致的分离:模型通常能捕捉诗意,但在开放设置中难以生成精确诗句补全,而基于识别的设置显著缩小了这一差距。在英语十四行诗上的平行实验显示出明显更强的补全性能,表明这些限制更多与训练暴露差异相关,而非固有架构约束。我们的发现强调了需要联合评估意义、形式及对文化重要文本的线索依赖访问的评估框架。GhazalBench可在该https URL获取。
英文摘要:
Persian poetry plays an active role in Iranian cultural practice, where verses by canonical poets such as Hafez and Saadi are frequently quoted, paraphrased, or completed from incomplete cues. Supporting such interactions requires language models to reliably access canonical verses from semantic and lexical information. We introduce GhazalBench, a benchmark for evaluating how large language models (LLMs) access the canonical surface forms of Persian ghazal under usage-grounded conditions. Unlike prior work that primarily treats memorization as a liability, GhazalBench studies settings in which access to exact wording is functionally important. The benchmark evaluates completion and recognition under varied semantic and lexical cues. Across proprietary and open-weight multilingual LLMs, exact completion remains challenging, while recognition is substantially stronger. Performance also varies across poets and model families. Parallel experiments on Shakespearean sonnets yield markedly higher completion for several models, consistent with differences in exposure to canonical texts. {Additional experiments suggest that post-training may reduce direct continuation-based access to canonical text.} Our findings motivate evaluation frameworks that distinguish production from recognition and assess access to culturally significant canonical texts. GhazalBench is available at https://github.com/kalhorghazal/GhazalBench.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。