TW-LegalBench: 衡量台湾法律理解
TW-LegalBench: Measuring Taiwanese Legal Understanding
AI总结：
提出TW-LegalBench基准，包含多项选择、开放式问答和法律判决预测任务，评估13个LLM在台湾法律上的表现，发现顶尖模型通过律师考试但未达到法官检察官标准，且法律条文引用困难。
摘要：
大型语言模型（LLM）在多种任务上展现出令人印象深刻的能力，但其在特定司法管辖区法律推理上的表现仍未充分探索。我们提出TW-LegalBench，利用台湾法律系统丰富的官方公开语料库，填补了在普通法基准（侧重英文来源）和大陆法基准（侧重简体中文来源）之外评估LLM在台湾法律上的空白。TW-LegalBench包含三种任务类型：（1）涵盖18个专业领域五年官方考试的超过16,000道多项选择题（MCQ）；（2）来自法律专业人员考试的117道开放式问答题（OEQ），附有官方评分标准；（3）超过14,000个法律判决预测（LJP）实例，涵盖数百种犯罪类别。我们使用MCQ的准确率、基于评分标准点的分解式LLM作为裁判框架评估OEQ，以及LJP的判决准确性和法条引用指标，评估了13个LLM。我们的结果显示，表现最佳的模型超过了合格律师的通过门槛（通过率：11%），但未达到法官和检察官的通过标准（通过率：1-2%）。对于LJP，虽然模型展示了合理的判决类型准确性和刑期预测能力，但它们难以准确引用具体法律条文。这些发现表明，即使LLM在资格考试上的表现接近人类水平，可靠的 legal 文本生成仍然具有挑战性。
英文摘要：
Large language models (LLMs) have shown impressive capabilities across diverse tasks, yet their performance on jurisdiction-specific legal reasoning remains underexplored. We present TW-LegalBench that utilizes Taiwanese legal system's rich official corpus open to the public to fill the gap in evaluating LLMs on Taiwanese law, among common-law benchmarks that focus on English sources and civil-law benchmarks focusing on sources of Simplified Chinese. TW-LegalBench comprises three task types: (1) over 16,000 multiple-choice questions (MCQs) across five years of official examinations in 18 professional domains; (2) 117 open-ended essay questions (OEQs) from examinations for legal professionals with official scoring rubrics; and (3) more than 14,000 legal judgment prediction (LJP) instances covering hundreds of crime categories. We evaluate 13 LLMs using accuracy for MCQs, a decomposed LLM-as-Judge framework based on the scoring rubric points for OEQs, and metrics for sentencing accuracy and statute citation for LJP. Our results reveal that top-performing models exceed the passing threshold for qualified lawyers (passing rate: 11%) but fall short of that for judges and prosecutors (passing rate: 1~2%). For LJP, while models demonstrate reasonable verdict type accuracy and sentence prediction capability, they struggle to cite exact legal articles. These findings highlight that reliable legal text generation remains challenging for LLMs, even though their performance on qualification examinations approaches human level.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
