arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.18645cs.CL
收藏

选择性预测的技艺:面向可靠的案件结果分类——基于欧洲人权法院案件的实证研究

The Craft of Selective Prediction: Towards Reliable Case Outcome Classification -- An Empirical Study on European Court of Human Rights Cases

School of Computation, Information, and Technology(计算、信息与技术学院);Technical University of Munich(慕尼黑工业大学)

T. Y. S. S. Santosh, Irtiza Chowdhury, Shanshan Xu, Matthias Grabmair

AI总结:

本文首次在法律NLP的选择性预测框架下,实证研究了预训练语料、置信度估计器和微调损失对案件结果分类模型可靠性的影响,发现特定领域预训练语料、Monte Carlo dropout和置信错误正则化能有效提升模型校准并缓解过度自信。

摘要:

在法律NLP的高风险决策任务中,如案件结果分类(COC),量化模型的预测置信度至关重要。置信度估计使人类能够做出更明智的决策,特别是在模型确定性较低或错误后果严重的情况下。然而,现有的大多数COC研究优先考虑高任务性能而非模型可靠性。本文在选择性预测框架内,实证研究了预训练语料库、置信度估计器和微调损失等各种设计选择如何影响COC模型的可靠性。我们在多标签COC任务上的实验(针对欧洲人权法院(ECtHR)案件)突出了多样化且特定领域的预训练语料库对于更好校准的重要性。此外,我们证明较大的模型往往表现出过度自信,Monte Carlo dropout方法产生可靠的置信度估计,而置信错误正则化有效缓解了过度自信。据我们所知,这是法律NLP中首次对选择性预测进行系统探索。我们的发现强调,需要进一步研究以增强置信度测量并提升法律领域模型的可信度。

英文摘要:

In high-stakes decision-making tasks within legal NLP, such as Case Outcome Classification (COC), quantifying a model's predictive confidence is crucial. Confidence estimation enables humans to make more informed decisions, particularly when the model's certainty is low, or where the consequences of a mistake are significant. However, most existing COC works prioritize high task performance over model reliability. This paper conducts an empirical investigation into how various design choices including pre-training corpus, confidence estimator and fine-tuning loss affect the reliability of COC models within the framework of selective prediction. Our experiments on the multi-label COC task, focusing on European Court of Human Rights (ECtHR) cases, highlight the importance of a diverse yet domain-specific pre-training corpus for better calibration. Additionally, we demonstrate that larger models tend to exhibit overconfidence, Monte Carlo dropout methods produce reliable confidence estimates, and confident error regularization effectively mitigates overconfidence. To our knowledge, this is the first systematic exploration of selective prediction in legal NLP. Our findings underscore the need for further research on enhancing confidence measurement and improving the trustworthiness of models in the legal domain.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑