LegalRikai:开放基准——复杂日本企业法律任务的基准
LegalRikai: Open Benchmark -- Benchmark for Complex Japanese Corporate Legal Tasks
AI总结:
LegalRikai基准通过四个复杂任务评估日本企业法律实践,揭示模型在文档编辑中的不足,并提出数据集评估框架以推动法律领域实践研究。
摘要:
本文介绍了LegalRikai:开放基准,一个新的包含四个复杂任务的基准,模拟日本企业法律实践。该基准由法律专业人士在律师的监督下创建。该基准包含100个需要长格式、结构化输出的样本,并针对多个实际标准进行了评估。我们使用领先的LLM进行了人类和自动化评估,包括GPT-5、Gemini 2.5 Pro和Claude Opus 4.1。我们的人类评估发现,抽象指令导致了不必要的修改,突显了模型在文档级编辑方面的弱点,这些弱点在传统短文本任务中被忽视。此外,我们的分析表明,自动化评估在具有明确语言基础的标准上与人类判断一致,但评估结构一致性仍是一个挑战。结果表明,自动化评估在专家资源有限时可以作为筛选工具。我们提出了一种数据集评估框架,以促进法律领域更具实践导向的研究。
英文摘要:
This paper introduces LegalRikai: Open Benchmark, a new benchmark comprising four complex tasks that emulate Japanese corporate legal practices. The benchmark was created by legal professionals under the supervision of an attorney. This benchmark has 100 samples that require long-form, structured outputs, and we evaluated them against multiple practical criteria. We conducted both human and automated evaluations using leading LLMs, including GPT-5, Gemini 2.5 Pro, and Claude Opus 4.1. Our human evaluation revealed that abstract instructions prompted unnecessary modifications, highlighting model weaknesses in document-level editing that were missed by conventional short-text tasks. Furthermore, our analysis reveals that automated evaluation aligns well with human judgment on criteria with clear linguistic grounding, and assessing structural consistency remains a challenge. The result demonstrates the utility of automated evaluation as a screening tool when expert availability is limited. We propose a dataset evaluation framework to promote more practice-oriented research in the legal domain.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。