本文首次在法律NLP的选择性预测框架下,实证研究了预训练语料、置信度估计器和微调损失对案件结果分类模型可靠性的影响,发现特定领域预训练语料、Monte Carlo dropout和置信错误正则化能有效提升模型校准并缓解过度自信。
摘要:
在法律NLP的高风险决策任务中,如案件结果分类(COC),量化模型的预测置信度至关重要。置信度估计使人类能够做出更明智的决策,特别是在模型确定性较低或错误后果严重的情况下。然而,现有的大多数COC研究优先考虑高任务性能而非模型可靠性。本文在选择性预测框架内,实证研究了预训练语料库、置信度估计器和微调损失等各种设计选择如何影响COC模型的可靠性。我们在多标签COC任务上的实验(针对欧洲人权法院(ECtHR)案件)突出了多样化且特定领域的预训练语料库对于更好校准的重要性。此外,我们证明较大的模型往往表现出过度自信,Monte Carlo dropout方法产生可靠的置信度估计,而置信错误正则化有效缓解了过度自信。据我们所知,这是法律NLP中首次对选择性预测进行系统探索。我们的发现强调,需要进一步研究以增强置信度测量并提升法律领域模型的可信度。
英文摘要:
In high-stakes decision-making tasks within legal NLP, such as Case Outcome Classification (COC), quantifying a model's predictive confidence is crucial. Confidence estimation enables humans to make more informed decisions, particularly when the model's certainty is low, or where the consequences of a mistake are significant. However, most existing COC works prioritize high task performance over model reliability. This paper conducts an empirical investigation into how various design choices including pre-training corpus, confidence estimator and fine-tuning loss affect the reliability of COC models within the framework of selective prediction. Our experiments on the multi-label COC task, focusing on European Court of Human Rights (ECtHR) cases, highlight the importance of a diverse yet domain-specific pre-training corpus for better calibration. Additionally, we demonstrate that larger models tend to exhibit overconfidence, Monte Carlo dropout methods produce reliable confidence estimates, and confident error regularization effectively mitigates overconfidence. To our knowledge, this is the first systematic exploration of selective prediction in legal NLP. Our findings underscore the need for further research on enhancing confidence measurement and improving the trustworthiness of models in the legal domain.
LexGenie: Automated Generation of Structured Reports for European Court of Human Rights Case Law
School of Computation, Information, and Technology(计算、信息与技术学院;慕尼黑工业大学);Technical University of Munich(国际与发展研究研究生院);Graduate Institute of International and Development Studies
Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases
EY AI Centre of Excellence in Document Intelligence(安永文档智能人工智能卓越中心);NCSR “Demokritos”(希腊德谟克利特国家科学研究中心);Athens University of Economics and Business(雅典经济与商业大学);University of Sheffield(谢菲尔德大学);University of York(约克大学)
VECHR: A Dataset for Explainable and Robust Classification of Vulnerability Type in the European Court of Human Rights
Technical University of Munich(慕尼黑工业大学);LMU Munich(慕尼黑大学);Graduate Institute of International and Development Studies(日内瓦高级国际关系及发展学院);University of Zürich(苏黎世大学)
LaCour!: Enabling Research on Argumentation in Hearings of the European Court of Human Rights
Technical University of Darmstadt(达姆施塔特工业大学);Ruhr University Bochum(波鸿鲁尔大学);University Alliance Ruhr(鲁尔大学联盟);Trustworthy Human Language Technologies(可信人类语言技术中心);Research Center Trustworthy Data Science and Security(可信数据科学与安全研究中心)
Combining topic modelling and citation network analysis to study case law from the European Court on Human Rights on the right to respect for private and family life
Can LLMs Reason in a Legally Meaningful Manner? A Small-scale Study on European Court of Human Rights Cases
University of Copenhagen(哥本哈根大学);Faculty of Law, University of Copenhagen(哥本哈根大学法学院);Department of Computer Science, University of Copenhagen(哥本哈根大学计算机科学系)
Financial University of Russia(俄罗斯联邦政府财政金融大学);MEPhI(莫斯科国立核研究大学);University Politehnica of Bucharest(布加勒斯特理工大学);University of Szeged(塞格德大学);Notre Dame College, Dhaka(达卡圣母学院);Mozaik Education(Mozaik教育)
Legal Experts Disagree With Rationale Extraction Techniques for Explaining ECtHR Case Outcome Classification
Trustworthy Human Language Technologies, Research Center Trustworthy Data Science and Security of the University Alliance Ruhr, Ruhr University Bochum(鲁尔大学联盟可信数据科学与安全研究中心,可信人类语言技术,波鸿鲁尔大学);Center for Critical Computational Studies, Goethe University Frankfurt(法兰克福歌德大学关键计算研究中心)
Through the Lens of Split Vote: Exploring Disagreement, Difficulty and Calibration in Legal Case Outcome Classification
Technical University of Munich(慕尼黑工业大学);Graduate Institute of International and Development Studies(国际与发展研究研究生院);IT University of Copenhagen(哥本哈根IT大学);LMU Munich(慕尼黑大学);Munich Center for Machine Learning (MCML)(慕尼黑机器学习中心);ELTEMATE