法庭中的谄媚者：大型语言模型（LLMs）是否对司法权威与演变的法律标准脆弱？
Sycophants in the Courtroom: Are LLMs Fragile to Juridical Authority and Evolving Legal Standards?
浏览论文内容
中文总结 AI 辅助
该研究通过比较诊断框架对比LLMs在法律与医学领域的表现，发现法律LLMs对司法权威扰动更脆弱，过度信任权威虚假信息，模型规模会放大该问题。
中文摘要 AI 辅助
在医学领域，主张若有基于稳定生物现实的经验证据支持则依然有效。相比之下，在法律领域，真相是偶然的，由司法管辖区、时间有效性以及权威来源的层级来定义。大型语言模型（LLMs）在医学执照考试中的近期成功，催生了其具备同等法律能力的期望。然而，这种类比掩盖了领域间的关键区别：与医学不同，法律表现往往较少依赖推理，而更多取决于何时适用、有效且无矛盾的外部权威。我们引入一个比较诊断框架，沿四个维度（知识回忆、依据、置信度与鲁棒性）将法律推理与医学基线进行评估，在应用于编码时间有效性与规范关系的新基准时，发现了显著的领域不对称性。医学LLMs可靠地从验证来源中获益，而法律LLMs则难以评估检索到的引用何时有用或具有误导性，在受扰动的环境中表现出过度自信，并对表面格式线索敏感。更大的模型规模会放大这种倾向，表明更强的指令遵循能力可能与对权威扰动的抵抗力减弱同时出现。这些发现显示，LLMs将法律视为非结构化文本而非具有约束力的先例，同时揭示了当外部参考与模型内部知识冲突时，存在过度信任权威但虚假信息的倾向。
英文摘要
In medicine, claims remain valid when supported by empirical evidence grounded in stable biological reality. In law, by contrast, truth is contingent, defined by jurisdiction, temporal validity, and the hierarchy of authoritative sources. The recent success of large language models (LLMs) on medical licensing examinations has encouraged an expectation of comparable legal competence. This analogy, however, obscures a critical distinction between domains. Unlike in medicine, legal performance often depends less on inference than on determining when external authority is applicable, valid, and non-contradictory. We introduce a comparative diagnostic framework evaluating legal reasoning against medical baselines along four axes (knowledge recall, grounding, confidence, and robustness), uncovering a sharp domain asymmetry when applied to a new benchmark that encodes temporal validity and normative relationships. While medical LLMs reliably benefit from verified sources, legal LLMs struggle to assess when retrieved citations are useful or misleading, exhibiting overconfidence in perturbed contexts and sensitivity to superficial formatting cues. Increased model scale amplifies this tendency, revealing that stronger instruction following can coincide with weaker resistance to authoritative perturbations. These findings show that LLMs treat law as unstructured text rather than binding precedent, while revealing a tendency to over-trust authoritative but false information when external references conflict with a model's internal knowledge.
发表机构
- University of Bologna（博洛尼亚大学）
机构由 AI 辅助整理，请以论文原文为准。
