Swiss-Bench SBP-002：瑞士法律与监管任务上的前沿模型比较
Swiss-Bench SBP-002: A Frontier Model Comparison on Swiss Legal and Regulatory Tasks
AI总结：
本文提出Swiss-Bench SBP-002，一个包含395个专家定制任务的三语基准，评估十种前沿模型在瑞士监管任务中的表现，揭示了不同任务类型的难度差异及模型性能分布。
摘要：
尽管近期工作已对瑞士法律翻译和学术法律推理进行了基准测试，但现有基准未评估前沿模型在应用瑞士监管合规任务中的表现。本文介绍了Swiss-Bench SBP-002，一个包含395个专家定制任务的三语基准，涵盖三个瑞士监管领域（FINMA、Legal-CH、EFK）、七种任务类型和三种语言（德语、法语、意大利语）。通过结构化的三维评分框架，使用盲评三法官LLM小组（GPT-4o、Claude Sonnet 4、Qwen3-235B）进行评估，采用多数投票聚合和加权Kappa=0.605，参考答案由独立法律专家在100个子集上验证（73%评为正确，0%错误，完美法律准确性）。结果揭示了三个性能集群：Tier A（35-38%正确）、Tier B（26-29%）和Tier C（13-21%）。该基准证明困难：即使排名第一的模型（Qwen 3.5 Plus）也仅达到38.2%正确率，47.3%错误，14.4%部分正确。任务类型难度差异很大：法律翻译和案例分析正确率在69-72%，而监管问答、幻觉检测和差距分析仍低于9%。在该名单（七种开放权重，三种闭源）中，开放权重模型领先排名，几个开放权重模型表现或超过其闭源对应物。这些发现为在零检索条件下评估前沿模型在瑞士监管任务中的能力提供了初始实证参考点。
英文摘要：
While recent work has benchmarked large language models on Swiss legal translation (Niklaus et al., 2025) and academic legal reasoning from university exams (Fan et al., 2025), no existing benchmark evaluates frontier model performance on applied Swiss regulatory compliance tasks. I introduce Swiss-Bench SBP-002, a trilingual benchmark of 395 expert-crafted items spanning three Swiss regulatory domains (FINMA, Legal-CH, EFK), seven task types, and three languages (German, French, Italian), and evaluate ten frontier models from March 2026 using a structured three-dimension scoring framework assessed via a blind three-judge LLM panel (GPT-4o, Claude Sonnet 4, Qwen3-235B) with majority-vote aggregation and weighted kappa = 0.605, with reference answers validated by an independent human legal expert on a 100-item subset (73% rated Correct, 0% Incorrect, perfect Legal Accuracy). Results reveal three descriptive performance clusters: Tier A (35-38% correct), Tier B (26-29%), and Tier C (13-21%). The benchmark proves difficult: even the top-ranked model (Qwen 3.5 Plus) achieves only 38.2% correct, with 47.3% incorrect and 14.4% partially correct. Task type difficulty varies widely: legal translation and case analysis yield 69-72% correct rates, while regulatory Q&A, hallucination detection, and gap analysis remain below 9%. Within this roster (seven open-weight, three closed-source), an open-weight model leads the ranking, and several open-weight models match or outperform their closed-source counterparts. These findings provide an initial empirical reference point for assessing frontier model capability on Swiss regulatory tasks under zero-retrieval conditions.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
