arXiv 2410.105422024-10-14cs.CLcs.AIcs.IRcs.LG

大语言模型时代现实场景中法律判决预测的重新思考

Rethinking Legal Judgement Prediction in a Realistic Scenario in the Era of Large Language Models

Indian Institute of Technology Kanpur (IIT-K)（印度理工学院坎普尔分校）；Indian Institute of Technology Kharagpur (IIT-Kgp)（印度理工学院卡拉格普尔分校）

Shubham Kumar Nigam, Aniket Deroy, Subhankar Maity, Arnab Bhattacharya

备注：: Accepted on NLLP at EMNLP 2024

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究在印度判决背景下，利用Transformer模型和LLM在现实场景中预测法律判决，发现GPT-3.5 Turbo表现稳健且结合法规与判例可显著提升效果，但LLM尚未达到专家级水平。

摘要：

本研究在印度判决的背景下，利用一系列基于Transformer的模型（包括InLegalBERT、BERT和XLNet）以及Llama-2和GPT-3.5 Turbo等LLM，调查了现实场景中的判决预测。在这种现实场景中，我们模拟了在案件提交法庭裁决时如何预测判决，仅使用当时可用的信息，例如案件事实、法规、判例和论据。这种方法模仿了现实世界的条件，即必须在没有事后诸葛亮优势的情况下做出决定，这与以往研究中常见的回顾性分析不同。对于Transformer模型，我们尝试使用分层Transformer和判决事实摘要来优化这些模型的输入。我们对LLM的实验表明，GPT-3.5 Turbo在现实场景中表现出色，在判决预测中展现出稳健的性能。此外，结合额外的法律信息（如法规和判例）显著改善了预测任务的结果。LLM还为其预测提供了解释。为了评估这些预测和解释的质量，我们引入了两个人工评估指标：Clarity和Linking。我们从自动评估和人工评估中的发现表明，尽管LLM取得了进步，但它们在判决预测和解释任务中尚未达到专家级性能。

英文摘要：

This study investigates judgment prediction in a realistic scenario within the context of Indian judgments, utilizing a range of transformer-based models, including InLegalBERT, BERT, and XLNet, alongside LLMs such as Llama-2 and GPT-3.5 Turbo. In this realistic scenario, we simulate how judgments are predicted at the point when a case is presented for a decision in court, using only the information available at that time, such as the facts of the case, statutes, precedents, and arguments. This approach mimics real-world conditions, where decisions must be made without the benefit of hindsight, unlike retrospective analyses often found in previous studies. For transformer models, we experiment with hierarchical transformers and the summarization of judgment facts to optimize input for these models. Our experiments with LLMs reveal that GPT-3.5 Turbo excels in realistic scenarios, demonstrating robust performance in judgment prediction. Furthermore, incorporating additional legal information, such as statutes and precedents, significantly improves the outcome of the prediction task. The LLMs also provide explanations for their predictions. To evaluate the quality of these predictions and explanations, we introduce two human evaluation metrics: Clarity and Linking. Our findings from both automatic and human evaluations indicate that, despite advancements in LLMs, they are yet to achieve expert-level performance in judgment prediction and explanation tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。