衡量忠实性与弃权（不执行）：一种用于评估LLM生成的三层基于案例的法律论证的自动化流水线
Measuring Faithfulness and Abstention: An Automated Pipeline for Evaluating LLM-Generated 3-ply Case-Based Legal Arguments
AI总结：
本文提出一种自动化流水线，通过外部LLM提取因子并与案件三元组比对，评估LLM生成三层法律论证时的忠实性、因子利用和弃权（不执行）能力，发现现有模型虽幻觉率低但因子利用不足且弃权表现差。
摘要：
大型语言模型（LLMs）在诸如论证生成等复杂法律任务中展现出潜力，但其可靠性仍然是一个值得关注的问题。本文在先前使用人工评估来评估LLM生成三层法律论证的试点工作基础上，引入了一种自动化流水线来评估LLM在该任务上的表现，特别关注忠实性（不存在幻觉）、因子利用以及适当的弃权（不执行）。我们将幻觉定义为生成输入案件材料中不存在的因子，将弃权（不执行）定义为模型在收到指令且不存在事实依据时能够克制不生成论证的能力。我们的自动化方法使用一个外部LLM从生成的论证中提取因子，并将其与输入案件三元组（当前案件和两个先例案件）中提供的地面真值因子进行比较。我们在三个难度递增的测试上评估了八个不同的LLM：1）生成一个标准的三层论证，2）在先例角色互换的情况下生成论证，以及3）识别由于缺乏共享因子而无法生成论证并进行弃权（不执行）。我们的研究结果表明，尽管当前的LLM在可行的论证生成测试（测试1和测试2）中避免幻觉方面达到了较高的准确率（超过90%），但它们往往未能利用案件中存在的全部相关因子。关键的是，在弃权（不执行）测试（测试3）中，大多数模型未能遵循停止指令，反而在缺乏共同因子的情况下生成了虚假论证。该自动化流水线为评估这些关键的LLM行为提供了一种可扩展的方法，凸显了在可靠部署于法律环境之前，需要在因子利用和稳健的弃权（不执行）能力方面进行改进。链接：https://lizhang-aiandlaw.github.io/An-Automated-Pipeline-for-Evaluating-LLM-Generated-3-ply-Case-Based-Legal-Arguments/
英文摘要：
Large Language Models (LLMs) demonstrate potential in complex legal tasks like argument generation, yet their reliability remains a concern. Building upon pilot work assessing LLM generation of 3-ply legal arguments using human evaluation, this paper introduces an automated pipeline to evaluate LLM performance on this task, specifically focusing on faithfulness (absence of hallucination), factor utilization, and appropriate abstention. We define hallucination as the generation of factors not present in the input case materials and abstention as the model's ability to refrain from generating arguments when instructed and no factual basis exists. Our automated method employs an external LLM to extract factors from generated arguments and compares them against the ground-truth factors provided in the input case triples (current case and two precedent cases). We evaluated eight distinct LLMs on three tests of increasing difficulty: 1) generating a standard 3-ply argument, 2) generating an argument with swapped precedent roles, and 3) recognizing the impossibility of argument generation due to lack of shared factors and abstaining. Our findings indicate that while current LLMs achieve high accuracy (over 90%) in avoiding hallucination on viable argument generation tests (Tests 1 & 2), they often fail to utilize the full set of relevant factors present in the cases. Critically, on the abstention test (Test 3), most models failed to follow instructions to stop, instead generating spurious arguments despite the lack of common factors. This automated pipeline provides a scalable method for assessing these crucial LLM behaviors, highlighting the need for improvements in factor utilization and robust abstention capabilities before reliable deployment in legal settings. Link: https://lizhang-aiandlaw.github.io/An-Automated-Pipeline-for-Evaluating-LLM-Generated-3-ply-Case-Based-Legal-Arguments/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
