大语言模型的幽默能力评估：大喜利（Oogiri）生成与评价的多维度分析
Assessing the Capabilities of LLMs in Humor:A Multi-dimensional Analysis of Oogiri Generation and Evaluation
AI总结：
本文针对现有大语言模型幽默能力单维度评估的不足，以日本大喜利游戏为载体构建六维度标注数据集，评估LLM的幽默生成与评价能力，发现其存在共情缺失缺陷并开放语料。
摘要：
计算幽默是打造高级、有吸引力的自然语言处理（NLP）应用（如复杂对话系统）的前沿方向。此前已有研究对大语言模型（LLM）的幽默能力开展基准测试，但这类研究往往依赖单维度评价，例如仅判断内容是否“好笑”。本文认为有必要对幽默形成多层面的理解，并通过日本即兴喜剧游戏形式——大喜利（Oogiri）的视角系统评估LLM，以填补该研究空白。为实现这一目标，我们基于现有Oogiri数据集补充了新来源的数据，还将LLM生成的Oogiri回复加入数据集扩充了语料规模。随后我们对扩充后的语料进行人工标注，从新颖性、清晰度、相关性、智性、共情性、整体好笑度六个维度采用5分制绝对评分。基于该数据集，我们评估了当前最优LLM在两项核心任务上的能力：一是生成有创意的Oogiri回复的能力，二是采用六维度评价体系判定回复好笑度的能力。结果显示，LLM生成回复的水平介于人类低水平表现与中等水平表现之间，但存在明显的共情性缺失问题，这一缺陷是其无法复现人类幽默评价结果的重要原因。对人类与模型评价数据的相关性分析进一步揭示二者评价标准存在根本分歧：LLM优先看重新颖性，而人类优先看重共情性。我们将标注语料向社区开放，为开发具备更高情绪智能、更成熟的对话智能体铺平道路。
英文摘要：
Computational humor is a frontier for creating advanced and engaging natural language processing (NLP) applications, such as sophisticated dialogue systems. While previous studies have benchmarked the humor capabilities of Large Language Models (LLMs), they have often relied on single-dimensional evaluations, such as judging whether something is simply ``funny.'' This paper argues that a multifaceted understanding of humor is necessary and addresses this gap by systematically evaluating LLMs through the lens of Oogiri, a form of Japanese improvisational comedy games. To achieve this, we expanded upon existing Oogiri datasets with data from new sources and then augmented the collection with Oogiri responses generated by LLMs. We then manually annotated this expanded collection with 5-point absolute ratings across six dimensions: Novelty, Clarity, Relevance, Intelligence, Empathy, and Overall Funniness. Using this dataset, we assessed the capabilities of state-of-the-art LLMs on two core tasks: their ability to generate creative Oogiri responses and their ability to evaluate the funniness of responses using a six-dimensional evaluation. Our results show that while LLMs can generate responses at a level between low- and mid-tier human performance, they exhibit a notable lack of Empathy. This deficit in Empathy helps explain their failure to replicate human humor assessment. Correlation analyses of human and model evaluation data further reveal a fundamental divergence in evaluation criteria: LLMs prioritize Novelty, whereas humans prioritize Empathy. We release our annotated corpus to the community to pave the way for the development of more emotionally intelligent and sophisticated conversational agents.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
