POEMetric：人类的最后一段
POEMetric: The Last Stanza of Humanity
AI总结：
本文提出POEMetric框架，评估诗歌生成能力，通过人类和LLM的对比发现，尽管顶级模型在形式准确性和主题匹配上表现优异，但整体诗歌质量仍逊于人类诗人。
摘要：
大型语言模型（LLMs）能够创作诗歌，但它们与人类诗人相比还有多远？本文介绍POEMetric，首个综合诗歌评估框架，评估生成诗歌的基本指令遵循能力、创造力、词汇多样性、情感共鸣和意象使用等高级能力，以及整体诗歌质量和作者身份评估。我们构建了一个包含203首英语诗歌的数据集，这些诗歌有7种固定形式，并标注了节奏、韵律模式和主题。我们对30个LLM进行了诗歌生成实验，基于相同的形式和主题，共生成6,090首LLM诗歌。基于POEMetric，我们通过基于规则的评估和LLM作为评判者评估了人类诗人和LLM的表现，结果由人类专家验证。结果显示，尽管顶级模型在形式准确性（4.26/5.00，以Gemini-2.5-Pro为评判者；相同以下）和主题匹配（4.99）上表现优异，但所有模型都无法达到人类诗人的高级能力水平，后者在创造力（4.02）、独特性（3.95）、情感共鸣（4.06）和意象使用（4.49）以及文学手法（4.67）方面表现卓越。人类在整体诗歌质量上也击败了表现最佳的LLM（4.22 vs. 3.20）。因此，诗歌生成仍然是LLM的严峻挑战。数据和代码已发布在https://github.com/Bingru-Li/POEMetric。
英文摘要：
Large Language Models (LLMs) can compose poetry, but how far are they from human poets? In this paper, we introduce POEMetric, the first comprehensive framework for poetry evaluation, examining 1) basic instruction-following abilities in generating poems according to a certain form and theme, 2) advanced abilities of showing creativity, lexical diversity, and idiosyncrasy, evoking emotional resonance, and using imagery and literary devices, and 3) general appraisal of the overall poem quality and estimation of authorship. We curated a human poem dataset - 203 English poems of 7 fixed forms annotated with meter, rhyme patterns and themes - and experimented with 30 LLMs for poetry generation based on the same forms and themes of the human data, totaling 6,090 LLM poems. Based on POEMetric, we assessed the performance of both human poets and LLMs through rule-based evaluation and LLM-as-a-judge, whose results were validated by human experts. Results show that, though the top model achieved high form accuracy (4.26 out of 5.00, with Gemini-2.5-Pro as a judge; same below) and theme alignment (4.99), all models failed to reach the same level of advanced abilities as human poets, who achieved unparalleled creativity (4.02), idiosyncrasy (3.95), emotional resonance (4.06), and skillful use of imagery (4.49) and literary devices (4.67). Humans also defeated the best-performing LLM in overall poem quality (4.22 vs. 3.20). As such, poetry generation remains a formidable challenge for LLMs. Data and codes are released at https://github.com/Bingru-Li/POEMetric.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
