透过分歧投票的视角：探究法律案件结果分类中的分歧、难度与校准
Through the Lens of Split Vote: Exploring Disagreement, Difficulty and Calibration in Legal Case Outcome Classification
AI总结：
本文通过欧洲人权法院分歧投票数据构建SV-ECHR数据集，系统探究法律案件结果分类中人类与模型在难度感知和校准上的一致性，发现现有模型与法官投票分布对齐有限，呼吁在法律NLP中重视人类标签变异的校准研究。
摘要：
在法律判决中，当法官无法达成一致决定时会出现分歧投票（SV），这给必须在多样法律论证和意见中寻找路径的律师带来了困难。在高风险领域，理解人类与AI系统之间对感知难度的一致性对于建立信任至关重要。然而，现有的NLP校准方法侧重于分类器对预测性能的自我认知，并以人类多数类别作为衡量标准，忽视了固有的人类标签变异（HLV）。本文探讨了分歧投票作为可自然观察的人类分歧与价值多元化的体现。我们从欧洲人权法院（ECHR）收集了法官的投票分布，并提出了SV-ECHR，一个包含分歧投票信息的案件结果分类（COC）数据集。我们构建了一个带有分歧投票特定子类别的分歧分类体系。我们进一步评估了模型与人类之间感知难度的一致性，以及COC模型的置信度校准和人类校准。我们观察到与法官投票分布的一致性有限。据我们所知，这是法律NLP中首次对基于人类判断的校准进行系统性探索。我们的研究强调了在法律决策任务中考虑人类标签变异来测量和增强模型校准方面进行进一步研究的必要性。
英文摘要：
In legal decisions, split votes (SV) occur when judges cannot reach a unanimous decision, posing a difficulty for lawyers who must navigate diverse legal arguments and opinions. In high-stakes domains, understanding the alignment of perceived difficulty between humans and AI systems is crucial to build trust. However, existing NLP calibration methods focus on a classifier's awareness of predictive performance, measured against the human majority class, overlooking inherent human label variation (HLV). This paper explores split votes as naturally observable human disagreement and value pluralism. We collect judges' vote distributions from the European Court of Human Rights (ECHR), and present SV-ECHR, a case outcome classification (COC) dataset with SV information. We build a taxonomy of disagreement with SV-specific subcategories. We further assess the alignment of perceived difficulty between models and humans, as well as confidence- and human-calibration of COC models. We observe limited alignment with the judge vote distribution. To our knowledge, this is the first systematic exploration of calibration to human judgements in legal NLP. Our study underscores the necessity for further research on measuring and enhancing model calibration considering HLV in legal decision tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
