专家意见分歧时,模型失效:检测法国法院判决中的隐含法律引用
Where Experts Disagree, Models Fail: Detecting Implicit Legal Citations in French Court Decisions
AI总结:
本文研究法国法院判决中隐含法律引用的检测问题,通过专家标注的基准数据集发现模型失效与专家分歧相关,采用多模型共识提升精度,达到76%的无监督精度。
摘要:
应用于法律研究的计算方法有望大规模分析法律。我们从一个简单问题出发:法院多大比例隐含适用法规?这需要区分法律推理与语义相似性。我们聚焦法国民法典在初审法院判决中的隐含引用,引入由三位法律专家标注的1,015段落-条文对基准数据集。我们显示专家分歧预测模型失效。标注者间一致性中等(κ=0.33),43%的分歧涉及事实描述与法律推理的边界。我们的监督集成实现F1=0.70(77%准确率),但此数值掩盖了不对称性:68%的假阳性落在33%的分歧案例中。尽管有这些限制,将任务重新定义为top-k排名并利用多模型共识,在无监督设置下达到k=200时76%的精度。此外,剩余假阳性倾向于揭示法律上模糊的应用而非明显错误。
英文摘要:
Applying computational methods to law at scale requires separating genuine legal reasoning from surface similarity. We study this through a concrete task: detecting implicit citations of the French Civil Code, where a court applies a statutory rule without naming it (a post-hoc question about the reasoning a court actually used). We release a benchmark of 1,015 passage-article pairs annotated by three legal experts. Our central finding is that their disagreement is itself informative: the third of cases the experts dispute are where models fail. Our best ensemble reaches an F1 score of 0.70 overall. Yet, two-thirds of its false positives fall on those disputed cases, a concentration that holds across all ten models we evaluate. Expert disagreement thus signals uncertainty in the task and in its gold label, which aggregate metrics hide. This should not block useful tools, however: reframed as top-$k$ ranking with multi-model consensus, the same signals reach 76% precision for the top-200 candidates of the benchmark without supervision.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。