新型词的终结：超越n-gram新颖性作为文本创造力的度量
Death of the Novel(ty): Beyond n-Gram Novelty as a Metric for Textual Creativity
AI总结：
本文研究了n-gram新颖性作为文本创造力度量的局限性，发现其与专家判断的创造力正相关，但高新颖性表达未必具有创造力，且LLM在识别非实用表达方面表现欠佳。
摘要：
n-gram新颖性被广泛用于评估语言模型生成超出训练数据文本的能力。最近，它也被用作衡量文本创造力的指标。然而，关于创造力的理论工作表明，这种方法可能不足，因为它没有考虑到创造力的双重性质：新颖性（文本的原创性）和适当性（文本的合理性与实用性）。我们通过8618个专家作家对新颖性、实用性、合理性的近距离阅读注释，调查了这种创造力概念与n-gram新颖性之间的关系。我们发现，尽管n-gram新颖性与专家作家判断的创造力正相关，但大约91%的高四分位n-gram新颖性表达未被判断为有创造力，这提醒人们不要仅依赖n-gram新颖性。此外，与人类写作文本不同，开源LLM中更高的n-gram新颖性与更低的实用性相关。在一项探索性研究中，我们还确认，前沿封闭源模型比人类更不可能产生有创造力的表达。使用我们的数据集，我们测试了零样本、少样本和微调模型是否能够识别专家认为新颖的表达（写作的积极方面）或非实用的表达（消极方面）。总体而言，前沿LLM在性能上远高于随机，但仍有改进空间，特别是在识别非实用表达方面存在困难。我们还发现，LLM-as-a-Judge的新颖性评分在分布外数据集中与专家作家的偏好一致，比基于n-gram的指标更一致。
英文摘要：
N-gram novelty is widely used to evaluate language models' ability to generate text outside of their training data. More recently, it has also been adopted as a metric for measuring textual creativity. However, theoretical work on creativity suggests that this approach may be inadequate, as it does not account for creativity's dual nature: novelty (how original the text is) and appropriateness (how sensical and pragmatic it is). We investigate the relationship between this notion of creativity and n-gram novelty through 8,618 expert writer annotations of novelty, pragmaticality, and sensicality via close reading of human- and AI-generated text. We find that while n-gram novelty is positively associated with expert writer-judged creativity, approximately 91% of top-quartile n-gram novel expressions are not judged as creative, cautioning against relying on n-gram novelty alone. Furthermore, unlike in human-written text, higher n-gram novelty in open-source LLMs correlates with lower pragmaticality. In an exploratory study with frontier closed-source models, we additionally confirm that they are less likely to produce creative expressions than humans. Using our dataset, we test whether zero-shot, few-shot, and finetuned models are able to identify expressions perceived as novel by experts (a positive aspect of writing) or non-pragmatic (a negative aspect). Overall, frontier LLMs exhibit performance much higher than random but leave room for improvement, especially struggling to identify non-pragmatic expressions. We further find that LLM-as-a-Judge novelty ratings align with expert writer preferences in an out-of-distribution dataset, more so than an n-gram based metric.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
