探索大型语言模型与分层框架用于大型非结构化法律文档分类
Exploring Large Language Models and Hierarchical Frameworks for Classification of Large Unstructured Legal Documents
AI总结：
本文提出 MESc 分层框架，结合微调 LLM、多层嵌入、无监督聚类与 transformer 编码器处理超长非结构化法律文档，并在多国法律数据集上验证其判决预测性能。
摘要：
法律判决预测通常面临案件文档长达数万词且结构不统一的问题。从此类文档预测判决是一项具有挑战性的任务，在没有结构标注的文档上更是如此。我们针对这些大型法律文档的分类及其结构信息缺失问题，探索了一种基于深度学习的分层框架，用于判决预测，并将其命名为 MESc，即“Multi-stage Encoder-based Supervised with-clustering（基于多阶段编码器、带聚类的监督方法）”。具体而言，我们将文档划分为多个部分，从自定义微调的大型语言模型最后四层提取它们的嵌入，并尝试通过无监督聚类近似其结构。随后，我们将这些结构用于另一组 transformer 编码器层，以学习块间表示。我们分析了具有数十亿参数的大型语言模型（LLM），包括 GPT-Neo 和 GPT-J，与 MESc 分层框架结合时的适应性，并将其与它们在法律文本上的独立性能进行比较。我们还研究了它们的域内（法律）迁移学习能力，以及在 MESc 中组合其最后几层嵌入所产生的影响。我们使用来自印度、欧盟和美国的法律文档，在 ILDC 数据集和 LexGLUE 数据集的一个子集上，通过大量实验和消融研究测试了这些方法及其有效性。与以往最先进方法相比，我们的方法实现了至少约 2 个百分点的总体性能提升。
英文摘要：
Legal judgment prediction suffers from the problem of long case documents exceeding tens of thousands of words, in general, and having a non-uniform structure. Predicting judgments from such documents becomes a challenging task, more so on documents with no structural annotation. We explore the classification of these large legal documents and their lack of structural information with a deep-learning-based hierarchical framework which we call MESc; "Multi-stage Encoder-based Supervised with-clustering"; for judgment prediction. Specifically, we divide a document into parts to extract their embeddings from the last four layers of a custom fine-tuned Large Language Model, and try to approximate their structure through unsupervised clustering. Which we use in another set of transformer encoder layers to learn the inter-chunk representations. We analyze the adaptability of Large Language Models (LLMs) with multi-billion parameters (GPT-Neo, and GPT-J) with the hierarchical framework of MESc and compare them with their standalone performance on legal texts. We also study their intra-domain(legal) transfer learning capability and the impact of combining embeddings from their last layers in MESc. We test these methods and their effectiveness with extensive experiments and ablation studies on legal documents from India, the European Union, and the United States with the ILDC dataset and a subset of the LexGLUE dataset. Our approach achieves a minimum total performance gain of approximately 2 points over previous state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
