评估基于LLM的法律引文预测方法：领域特定预训练、微调还是RAG？一个基准与澳大利亚法律案例研究
Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study
AI总结：
本研究提出AusLaw引文基准，系统评估多种LLM方法在法律引文预测上的表现，发现混合检索增强策略最佳，但仍有近50%性能差距。
摘要：
大型语言模型（LLMs）已在多种法律任务中展现出强大潜力，然而法律引文预测问题仍未得到充分探索。其核心在于，该任务要求细粒度的上下文理解以及对相关立法或先例的精确识别。我们引入了AusLaw引文基准，这是一个包含55,000个澳大利亚法律实例和18,677条唯一引文的真实世界数据集，据我们所知，这是首个具有如此规模和范围的数据集。随后，我们对一系列解决方案进行了系统性的基准测试：（i）对通用和法律专业LLM的标准提示，（ii）使用通用和领域特定嵌入的仅检索流水线，（iii）监督微调，以及（iv）通过查询扩展、投票集成或重排序将LLM与检索增强相结合的多种混合策略。结果表明，通用和法律特定LLM均不足以作为独立解决方案，其性能接近于零。在任务特定数据集上进行指令微调（即使是通用开源LLM）是最佳性能解决方案之一。我们强调，数据库粒度以及嵌入类型在基于检索的方法中起着关键作用，其中利用训练好的重排序器的混合方法取得了最佳结果。尽管如此，仍有近50%的性能差距，这凸显了这一具有挑战性的基准作为未来法律领域研究严格测试平台的价值。
英文摘要：
Large Language Models (LLMs) have demonstrated strong potential across legal tasks, yet the problem of legal citation prediction remains under-explored. At its core, this task demands fine-grained contextual understanding and precise identification of relevant legislation or precedent. We introduce the AusLaw Citation Benchmark, a real-world dataset comprising 55k Australian legal instances and 18,677 unique citations which to the best of our knowledge is the first of its scale and scope. We then conduct a systematic benchmarking across a range of solutions: (i) standard prompting of both general and law-specialised LLMs, (ii) retrieval-only pipelines with both generic and domain-specific embeddings, (iii) supervised fine-tuning, and (iv) several hybrid strategies that combine LLMs with retrieval augmentation through query expansion, voting ensembles, or re-ranking. Results show that neither general nor law-specific LLMs suffice as stand-alone solutions, with performance near zero. Instruction tuning (of even a generic open-source LLM) on task-specific dataset is among the best performing solutions. We highlight that database granularity along with the type of embeddings play a critical role in retrieval-based approaches, with hybrid methods which utilise a trained re-ranker delivering the best results. Despite this, a performance gap of nearly 50% remains, underscoring the value of this challenging benchmark as a rigorous test-bed for future research in legal-domain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
