法律专家对解释ECtHR案件结果分类的推理提取技术持不同意见
Legal Experts Disagree With Rationale Extraction Techniques for Explaining ECtHR Case Outcome Classification
AI总结：
本文研究了欧洲人权法院判决结果预测任务，引入了新的ECtHR数据集，并提出比较分析框架评估推理提取技术的解释能力，发现模型预测违规的'原因'与法律专家存在显著差异。
摘要：
可解释性对大型语言模型在法律领域中的应用至关重要，尤其是在需要信任和透明度的场景中。该任务的核心是法律结果预测，即模型预测法院是否会认定某权利被侵犯。我们在此任务上使用欧洲人权法院（ECtHR）的判决决定，引入了一个精心编纂的ECtHR数据集，包含正例（违规）和负例（非违规）案件。现有研究提出了针对任务的特定方法和模型无关的技术来解释下游性能，但尚不清楚哪种技术最能解释法律结果预测。为此，我们提出了一种比较分析框架，用于评估模型无关的可解释性方法。我们专注于两种推理提取技术，通过输入中的简洁、可被人类理解的文本片段来解释模型输出。我们通过归一化充分性和全面性指标评估忠实度，并通过法律专家对提取理由的判断评估合理性。我们还评估了使用LLM作为法官的可行性，使用这些专家评估作为参考。在新的ECtHR数据集上的实验表明，尽管模型的忠实度得分较高，但模型预测违规的'原因'与法律专家存在显著差异。我们的实验代码已公开在https://github.com/trusthlt/IntEval。
英文摘要：
Interpretability is critical for applications of large language models (LLMs) in the legal domain, where trust and transparency are essential. A central NLP task in this setting is legal outcome prediction, where models forecast whether a court will find a violation of a given right. We study this task on decisions from the European Court of Human Rights (ECtHR), introducing a new ECtHR dataset with carefully curated positive (violation) and negative (non-violation) cases. Existing works propose both task-specific approaches and model-agnostic techniques to explain downstream performance, but it remains unclear which techniques best explain legal outcome prediction. To address this, we propose a comparative analysis framework for model-agnostic interpretability methods. We focus on two rationale extraction techniques that justify model outputs with concise, human-interpretable text fragments from the input. We evaluate faithfulness via normalized sufficiency and comprehensiveness metrics, and plausibility via legal expert judgments of the extracted rationales. We also assess the feasibility of using LLM-as-a-Judge, using these expert evaluations as reference. Our experiments on the new ECtHR dataset show that models' "reasons" for predicting violations differ substantially from those of legal experts, despite strong faithfulness scores. The source code of our experiments is publicly available at https://github.com/trusthlt/IntEval.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
