SoCRATES：跨领域和社会认知变异的前瞻性LLM调解的可靠自动化评估
SoCRATES: Towards Reliable Automated Evaluation of Proactive LLM Mediation across Domains and Socio-cognitive Variations
AI总结：
提出SoCRATES基准，通过多领域真实冲突场景和五维社会认知适应轴评估LLM调解员，使用主题定位评估器实现0.82的人类专家一致性，发现最强模型仅缩小约三分之一的未调解共识差距。
摘要：
评估LLM调解员仍然具有挑战性，因为调解是一个实时轨迹，由争议者不断变化的情感、意图和背景塑造。现有的测试平台依赖于少数专家撰写的领域，主要变化战略姿态，并对每个话题的每一轮进行评分，引入了离题噪声。我们引入了SoCRATES，一个用于在现实的多领域测试平台中评估前瞻性LLM调解员的基准。它通过一个跨八个领域的代理管道从真实冲突中构建场景，探测五个社会认知适应轴（战略姿态、参与者组成、历史长度、情感反应和文化身份），并通过主题定位评估器仅对推进每个话题的轮次进行评分。该评估器与人类专家的一致性达到0.82，是每轮基线的两倍以上。对八个前沿LLM的基准测试发现，即使是最强的调解员，在多样化和现实的测试平台下，也仅能缩小约三分之一的未调解共识差距，且性能因社会认知轴而异，突显出进步在于对不同条件的社会适应。
英文摘要：
Evaluating LLM mediators remains challenging, as mediation unfolds as a real-time trajectory shaped by disputants' shifting emotions, intentions, and context. Existing testbeds rely on a few expert-authored domains, vary mainly strategic posture, and score every turn against every topic, introducing off-topic noise. We introduce SoCRATES, a benchmark for evaluating proactive LLM mediators in realistic, multi-domain testbeds. It constructs scenarios from real conflicts through an agentic pipeline across eight domains, probes five socio-cognitive adaptation axes (strategic posture, party composition, history length, emotional reactivity, and cultural identity), and scores each topic only on the turns that advance it via a topic-localized evaluator. The evaluator reaches 0.82 alignment with human experts, more than doubling a per-turn baseline. Benchmarking eight frontier LLMs, we find that even the strongest mediator closes only about a third of the unmediated consensus gap under diverse and realistic testbeds, with performance varying sharply by socio-cognitive axis, highlighting that progress lies in social adaptation to diverse conditions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
