Oogiri-Master：通过Oogiri基准测试幽默理解
Oogiri-Master: Benchmarking Humor Understanding via Oogiri
AI总结：
Oogiri-Master通过Oogiri基准测试，利用大量候选回应和独立评分机制，评估LLMs的幽默理解能力，并揭示语言因素与趣味性之间的关系。
摘要：
幽默是测试大语言模型（LLMs）中类人创造性思维的重要测试场。我们使用日本创意回应游戏Oogiri来研究幽默，其中参与者会根据给定提示生成巧妙的回应，并提出以下研究问题：什么使这些回应对人类来说是有趣的？以往的工作只提供了有限可靠的答案方法。现有数据集每个提示包含的候选回应较少，评分过程中暴露了流行信号，且缺乏客观且可比较的趣味性度量标准。因此，我们引入Oogiri-Master和Oogiri-Corpus，它们是一个基准和数据集，旨在使LLMs的幽默理解评估更加严格。每个提示都配以约100个多样化的候选回应，并由约100名人类评委独立评分，而无需访问他人的评分，从而减少流行偏差并实现稳健的聚合。利用Oogiri-Corpus，我们对与趣味性相关的语言因素进行了定量分析，如文本长度、歧义性和不协调性解决，并推导出预测人类判断的客观度量标准。随后，我们在Oogiri-Master中对一系列LLMs和人类基线进行基准测试，证明了最先进的模型接近人类表现，并且洞察增强的提示方法提高了模型性能。我们的结果为评估和推进LLMs中的幽默理解提供了系统的基础。
英文摘要：
Humor is a salient testbed for human-like creative thinking in large language models (LLMs). We study humor using the Japanese creative response game Oogiri, in which participants produce witty responses to a given prompt, and ask the following research question: What makes such responses funny to humans? Previous work has offered only limited reliable means to answer this question. Existing datasets contain few candidate responses per prompt, expose popularity signals during ratings, and lack objective and comparable metrics for funniness. Thus, we introduce Oogiri-Master and Oogiri-Corpus, which are a benchmark and dataset designed to enable rigorous evaluation of humor understanding in LLMs. Each prompt is paired with approximately 100 diverse candidate responses, and funniness is rated independently by approximately 100 human judges without access to others' ratings, reducing popularity bias and enabling robust aggregation. Using Oogiri-Corpus, we conduct a quantitative analysis of the linguistic factors associated with funniness, such as text length, ambiguity, and incongruity resolution, and derive objective metrics for predicting human judgments. Subsequently, we benchmark a range of LLMs and human baselines in Oogiri-Master, demonstrating that state-of-the-art models approach human performance and that insight-augmented prompting improves the model performance. Our results provide a principled basis for evaluating and advancing humor understanding in LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
