LexRubric：面向开放式法律任务的基于评分指南的诊断基准
LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Tasks
AI总结：
提出LexRubric基准，包含649个中国法律咨询与司法考试实例及12,337条原子评分标准，通过六维框架评估LLM在开放式法律任务中的可靠性，发现当前模型仍面临挑战。
摘要：
随着大型语言模型（LLM）越来越多地应用于现实法律任务，评估其开放式法律响应的可靠性变得至关重要。这些任务需要上下文敏感的答案，且容错空间极小，因此需要能够识别响应质量失败具体原因的细粒度诊断评估。我们引入了LexRubric，一个基于评分指南的基准，用于评估开放式中文法律任务。LexRubric包含来自法律咨询和司法考试的649个实例，这些实例既反映了日常法律需求，也体现了专业法律推理，覆盖14个法律场景。此外，它还包含12,337条由专家编写的原子评分标准，这些标准组织在一个统一的六维框架下，能够跨任务和评估维度进行准确的评估和诊断分析。为了验证评估的可靠性，我们测试了多个评判模型，并将基于模型的评判与人类评判进行了比较。我们进一步在LexRubric上评估了18个近期通用和法律领域的LLM。结果表明，不同模型展现出不同的能力特征，且开放式法律问题对当前LLM仍然具有挑战性。数据可在以下网址获取：https://github.com/foggpoy/LexRubric。
英文摘要：
As large language models (LLMs) are increasingly applied to real-world legal tasks, evaluating the reliability of their open-ended legal responses has become essential. These tasks require context-sensitive answers and allow little room for error, motivating fine-grained and diagnostic evaluation that can identify specific sources of response quality failures. We introduce LexRubric, a rubric-based benchmark for evaluating open-ended Chinese legal tasks. LexRubric contains 649 instances from legal consultation and judicial examination, which reflect both everyday legal needs and professional legal reasoning and cover 14 legal scenarios. It further includes 12,337 expert-written atomic scoring criteria organized under a unified six-dimensional framework, enabling accurate evaluation and diagnostic analysis across tasks and evaluation dimensions. To validate the reliability of the evaluation, we test multiple judge models and compare model-based judgments with human judgments. We further evaluate 18 recent general and legal-domain LLMs on LexRubric. Results show that different models exhibit distinct capability profiles, and that open-ended legal tasks remain challenging for current LLMs. Data is available at: https://github.com/foggpoy/LexRubric.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
