中文竞技辩论数据集与基准

中文摘要 AI 辅助

辩论裁决需要跟踪论点在互动中的发展过程，然而现有数据集很少能将细粒度的辩论记录与在真实比赛中依据统一评分标准收集的专业评委评判相结合。我们引入了一个数据集和基准，用于评估大型语言模型在比赛、阶段和辩手三个层面上对中文竞技辩论的理解能力。我们组织了182场比赛，招募了120名专业评委，每场比赛由三位评委依据预定义的评分标准独立评判。在排除记录不完整的比赛后，该数据集包含148场比赛、2698个阶段和20542个交流单元，并附有手动验证的记录和切分。它保留了原始的阶段得分、比赛投票、最佳辩手选票和裁决理由。我们定义了三个任务：获胜倾向预测、阶段得分预测和最佳辩手预测。对多个大型语言模型的零样本评估显示，获胜预测的最高准确率为66.2%，模型阶段得分与人类平均评分之间的最高皮尔逊相关系数为0.250，最佳辩手预测的最高准确率为56.8%。该数据集和基准为研究大型语言模型对互动论证的理解及其与专业评委的一致性提供了一个测试平台。

英文摘要

Debate adjudication requires tracking how arguments develop through interaction, yet existing datasets rarely combine fine-grained debate transcripts with professional judgments collected during real competitions under a shared rubric. We introduce a dataset and benchmark for evaluating large language models' understanding of competitive Chinese-language debate at the match, stage, and speaker levels. We organized 182 matches and recruited 120 professional judges, with each match independently adjudicated by three judges using a predefined rubric. After excluding matches with incomplete records, the dataset contains 148 matches, 2,698 stages, and 20,542 exchange units, with manually verified transcripts and segmentation. It preserves original stage scores, match votes, best-debater ballots, and adjudication rationales. We define three tasks: winner-tendency prediction, stage-score prediction, and best-debater prediction. Zero-shot evaluation of multiple large language models yields a highest winner-prediction accuracy of 66.2%, a highest Pearson correlation of 0.250 between model stage scores and mean human ratings, and a highest best-debater prediction accuracy of 56.8%. The dataset and benchmark provide a testbed for studying large language models' understanding of interactive argumentation and their agreement with professional judges.

发表机构

• University of Auckland（奥克兰大学）
• South China Agricultural University（华南农业大学）
• Sanming University（三明学院）

机构由 AI 辅助整理，请以论文原文为准。

补充信息

备注

25 pages, 2 figures