arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2201.06653cs.LGcs.AIcs.CL
收藏

法律领域中以数据为中心的机器学习

Data-Centric Machine Learning in the Legal Domain

Université de Montréal(蒙特利尔大学);Carnegie Mellon University(卡内基梅隆大学);Hofstra University(霍夫斯特拉大学);University of Pittsburgh(匹兹堡大学)

Hannes Westermann, Jaromir Savelka, Vern R. Walker, Kevin D. Ashley, Karim Benyekhlef

摘要:

机器学习研究通常从过程早期创建的固定数据集开始。实验的重点是找到一个模型和训练过程,导致在一些选定的评估指标方面的最佳性能。本文探讨了数据集的变化如何影响模型的测量性能。我们使用三个来自法律领域的公开数据集,研究它们的大小、训练/测试分割以及人类标记准确性的变化如何影响经过训练的深度学习分类器的性能。我们评估整体表现(加权平均)以及每个类的表现。观察到的效果是令人惊讶的明显,特别是当考虑每类性能。我们研究如何“语义同质性”的一个类,即,句子在语义嵌入空间中的接近程度影响其分类的难度。所呈现的结果对人工智能和法律领域的数据汇集和管理工作具有深远的影响。结果还表明,可以考虑对数据集进行增强,以及ML模型的改进,作为提高AI & Law中各种任务分类性能的额外途径。最后,我们讨论了需要一个既定的方法来评估数据集属性的潜在影响。

英文摘要:

Machine learning research typically starts with a fixed data set created early in the process. The focus of the experiments is finding a model and training procedure that result in the best possible performance in terms of some selected evaluation metric. This paper explores how changes in a data set influence the measured performance of a model. Using three publicly available data sets from the legal domain, we investigate how changes to their size, the train/test splits, and the human labelling accuracy impact the performance of a trained deep learning classifier. We assess the overall performance (weighted average) as well as the per-class performance. The observed effects are surprisingly pronounced, especially when the per-class performance is considered. We investigate how "semantic homogeneity" of a class, i.e., the proximity of sentences in a semantic embedding space, influences the difficulty of its classification. The presented results have far reaching implications for efforts related to data collection and curation in the field of AI & Law. The results also indicate that enhancements to a data set could be considered, alongside the advancement of the ML models, as an additional path for increasing classification performance on various tasks in AI & Law. Finally, we discuss the need for an established methodology to assess the potential effects of data set properties.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑