什么时候培训有帮助？评估法律和CaseHOLD数据集的自监督学习
When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset
摘要：
虽然自监督学习在自然语言处理方面取得了快速进展，但研究人员何时应该进行资源密集型的特定领域预训练（领域预训练）仍不清楚。令人困惑的是，尽管法律语言被广泛认为是独特的，但法律几乎没有记录在案的领域预训练的实质性收益。我们假设，这些现有的结果源于这样一个事实，即现有的法律NLP任务太容易，无法满足领域预训练可以帮助的条件。为了解决这个问题，我们首先介绍了CaseHOLD（法律判决的案例持有），这是一个新的数据集，由超过53，000个多项选择题组成，用于识别引用案例的相关持有。该数据集为律师提供了一项基本任务，从NLP的角度来看，它既有法律意义，又很困难（F1为0.4，BiLSTM基线）。其次，我们评估了CaseHOLD和现有合法NLP数据集的性能增益。虽然Transformer架构（BERT）在一般语料库上进行预训练（Google图书和维基百科）提高性能，域预训练（使用美国所有法院的大约350万个判决的语料库，其规模大于BERT），使用自定义法律词汇表，通过CaseHOLD表现出最显著的性能提升（在F1上获得7.2%的收益，比BERT提高了12%），并在其他两项法律任务上获得了一致的性能收益。第三，我们表明，域预训练可能是必要的，当任务表现出足够的相似性的预训练语料库：在三个法律任务的性能增加的水平直接挂钩的任务的域特异性。我们的研究结果告知研究人员何时应该进行资源密集型的预训练，并表明基于transformer的架构也会学习暗示不同法律语言的嵌入。
英文摘要：
While self-supervised learning has made rapid advances in natural language processing, it remains unclear when researchers should engage in resource-intensive domain-specific pretraining (domain pretraining). The law, puzzlingly, has yielded few documented instances of substantial gains to domain pretraining in spite of the fact that legal language is widely seen to be unique. We hypothesize that these existing results stem from the fact that existing legal NLP tasks are too easy and fail to meet conditions for when domain pretraining can help. To address this, we first present CaseHOLD (Case Holdings On Legal Decisions), a new dataset comprised of over 53,000+ multiple choice questions to identify the relevant holding of a cited case. This dataset presents a fundamental task to lawyers and is both legally meaningful and difficult from an NLP perspective (F1 of 0.4 with a BiLSTM baseline). Second, we assess performance gains on CaseHOLD and existing legal NLP datasets. While a Transformer architecture (BERT) pretrained on a general corpus (Google Books and Wikipedia) improves performance, domain pretraining (using corpus of approximately 3.5M decisions across all courts in the U.S. that is larger than BERT's) with a custom legal vocabulary exhibits the most substantial performance gains with CaseHOLD (gain of 7.2% on F1, representing a 12% improvement on BERT) and consistent performance gains across two other legal tasks. Third, we show that domain pretraining may be warranted when the task exhibits sufficient similarity to the pretraining corpus: the level of performance increase in three legal tasks was directly tied to the domain specificity of the task. Our findings inform when researchers should engage resource-intensive pretraining and show that Transformer-based architectures, too, learn embeddings suggestive of distinct legal language.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
