SCRuB：基于规则评估的社会概念推理
SCRuB: Social Concept Reasoning under Rubric-Based Evaluation
AI总结：
本文提出SCRuB框架，用于评估大语言模型在社会概念推理方面的能力，通过专家和模型响应对比，展示前沿模型在五个维度上优于人类专家。
摘要：
尽管许多研究强调大语言模型（LLM）在数学或技术任务上的推理能力，但很少有研究关注社会概念的推理：塑造社会规范、文化和制度的抽象思想。这种未被充分研究的能力对于现代作为社会代理的模型至关重要，但目前缺乏系统的评估方法。我们引入SCRuB（基于规则评估的社会概念推理）框架，旨在衡量模型在社会概念推理上的深度和批判性。SCRuB分为三个阶段：从已有来源构建提示、由专家和模型生成响应、以及使用五维批判性思维评估标准进行比较评估。为了使流程具有通用性，我们引入了学科视角面板 ensemble，并通过独立专家法官验证。我们发布了SCRuBEval（n=4,711评估提示）和SCRuBAnnotations（300名专家撰写的响应和150名专家的比较判断，来自45名博士级学者）。我们的结果显示，前沿模型在所有五个评估维度上均优于人类专家。在1,170次配对比较中，专家法官将模型响应排在首位的有80.8%的判断，整体更喜欢模型响应的有74.4%的时间。最终，本研究提供了社会概念推理评估饱和的首次专家基础演示：单轮考试式格式对模型和人类而言均已达到极限。
英文摘要：
While many studies of Large Language Model (LLM) reasoning capabilities emphasize mathematical or technical tasks, few address reasoning about social concepts: the abstract ideas shaping social norms, culture, and institutions. This understudied capability is essential for modern models acting as social agents, yet no systematic evaluation methodology targets it. We introduce SCRuB (Social Concept Reasoning under Rubric-Based Evaluation), a framework designed for this setting of task indeterminacy. Our goal is to measure the degree to which a model reasons about social concepts with the depth and critical rigor of a human expert. SCRuB proceeds in three phases: prompt construction from established sources, response generation by experts and models, and comparative evaluation using a five-dimensional critical thinking rubric. To enable generalization of the pipeline, we introduce a Panel of Disciplinary Perspectives ensemble validated against independent expert judges. We release SCRuBEval (n=4,711 evaluation prompts) and SCRuBAnnotations (300 expert-authored responses and 150 expert comparative judgments from 45 PhD-level scholars). Our results show that frontier models consistently outperform human experts across all five rubric dimensions. Across 1,170 pairwise comparisons, expert judges ranked a model response first in 80.8% of judgments and preferred model responses overall 74.4% of the time. Ultimately, this study provides the first expert-grounded demonstration of evaluation saturation for social concept reasoning: the single-turn exam-style format has reached its ceiling for models and humans alike.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
