HumorRank: 基于锦标赛的大语言模型幽默生成评估排行榜
HumorRank: A Tournament-Based Leaderboard for Evaluating Humor Generation in Large Language Models
AI总结：
提出HumorRank，一种基于锦标赛的框架，通过理论指导的成对偏好判断对文本幽默生成进行排名，并利用Bradley-Terry估计生成全局排行榜。
摘要：
幽默在大语言模型（LLM）中仍然难以评估，因为一个回答是否有趣是主观的、比较性的，并且由相互作用的喜剧机制而非单一标量属性塑造。因此，现有的幽默评估协议往往产生孤立的分数或特定任务的判断，难以跨模型进行比较。我们引入了HumorRank，一种基于锦标赛的框架，通过理论指导的成对偏好判断对文本幽默生成进行排名。在SemEval-2026 MWAHAHA和Humor Transfer Bench上，HumorRank使用基于LLM的比较判断（基于言语幽默通论GTVH）评估了九个专有、开放权重和专门模型，并通过Bradley-Terry估计的锦标赛聚合生成全局排名。得到的排名跨评判者稳定：独立的Llama和Qwen LLM评判者在两个基准上均达到Kendall τ = 0.889。排行榜揭示了清晰的模型分层，表明强大的幽默生成不仅依赖于规模，还依赖于对喜剧机制（如不协调、简洁、升级和荒谬）的掌握。HumorRank提供了一种可扩展且可解释的方法，用于对LLM生成的幽默进行基准测试，而不完全依赖孤立的自动指标或有限的人工评估。
英文摘要：
Evaluating humor in large language models (LLMs) is an open challenge because existing approaches yield isolated, incomparable metrics rather than unified model rankings, making it difficult to track progress across systems. We introduce HumorRank, a tournament-based evaluation framework and leaderboard for textual humor generation. On two public benchmarks (SemEval-2026 MWAHAHA and Humor Transfer Bench), we conduct extensive automated pairwise evaluation across nine models spanning proprietary, open-weight, and specialized systems. Pairwise judgments are produced by LLM judges grounded in the General Theory of Verbal Humor (GTVH): each judge integrates structured comedic analysis into adjudication, jointly yielding a preference decision, an interpretable rationale, and mechanism, delivery, and failure tags rather than a black-box funniness score. Judgments are aggregated via an Adaptive Swiss tournament, with Bradley-Terry Maximum Likelihood Estimation (MLE) producing globally consistent humor generation capability rankings. Rankings are cross-judge stable: independent LLM judges (Llama 3.3 70B and Qwen 2.5 72B) yield Kendall tau = 0.889 on both benchmarks, and a human calibration study shows human-LLM agreement tracking human-human agreement on hard funny-versus-funny pairs. Our results demonstrate that HumorRank yields statistically grounded model stratifications, showing that humor quality is associated with mastery of comedic mechanisms such as incongruity, conciseness, escalation, and absurdity rather than model scale alone, with specialized fine-tuned models reaching parity with far larger systems. HumorRank thus provides a scalable, interpretable, and reproducible methodology for benchmarking and understanding LLM-generated humor.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
