利用大语言模型进行法律案例检索中的相关性判断
Leveraging Large Language Models for Relevance Judgments in Legal Case Retrieval
AI总结：
针对法律案例检索中相关性判断耗时且缺乏可解释性的问题，提出一种少样本方法，利用大语言模型分阶段模拟专家推理生成可解释判断，并验证了其可靠性及向小模型知识蒸馏的能力。
摘要：
确定哪些法律案例与给定查询相关，需要处理冗长的文本并应用细致的法律推理。传统上，这项任务需要耗费大量时间和具备领域专业知识，以识别关键的法律事实并得出合理的法学结论。此外，现有的包含法律案例相似性的数据通常缺乏可解释性，使得难以理解相关性判断背后的依据。随着大语言模型（LLMs）能力的不断提升，研究人员已开始探索其在该领域的潜力。尽管如此，如何采用通用大语言模型在法律案例检索中进行可靠的相关性判断，在很大程度上仍未被探索。为了填补这一研究空白，我们提出了一种新颖的少样本方法，由 LLMs 辅助生成与专家对齐且可解释的相关性判断。该方法将判断过程分解为多个阶段，模拟人类标注员的工作流程，并允许灵活引入专家推理以提高相关性判断的准确性。重要的是，它还确保了可解释的数据标注，为相关性评估过程提供了透明度和清晰度。通过对比 LLMs 与人类专家做出的相关性判断，我们实证表明所提出的方法能够产生可靠且有效的相关性评估。此外，我们证明，在极少的专家监督下，我们的方法能使大语言模型掌握案例分析专业知识，并随后通过基于标注的知识蒸馏将这一能力迁移到较小的模型中。
英文摘要：
Determining which legal cases are relevant to a given query involves navigating lengthy texts and applying nuanced legal reasoning. Traditionally, this task has demanded significant time and domain expertise to identify key Legal Facts and reach sound juridical conclusions. In addition, existing data with legal case similarities often lack interpretability, making it difficult to understand the rationale behind relevance judgments. With the growing capabilities of large language models (LLMs), researchers have begun investigating their potential in this domain. Nonetheless, the method of employing a general large language model for reliable relevance judgments in legal case retrieval remains largely unexplored. To address this gap in research, we propose a novel few-shot approach where LLMs assist in generating expert-aligned interpretable relevance judgments. The proposed approach decomposes the judgment process into several stages, mimicking the workflow of human annotators and allowing for the flexible incorporation of expert reasoning to improve the accuracy of relevance judgments. Importantly, it also ensures interpretable data labeling, providing transparency and clarity in the relevance assessment process. Through a comparison of relevance judgments made by LLMs and human experts, we empirically demonstrate that the proposed approach can yield reliable and valid relevance assessments. Furthermore, we demonstrate that with minimal expert supervision, our approach enables a large language model to acquire case analysis expertise and subsequently transfers this ability to a smaller model via annotation-based knowledge distillation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
