LLMs是否已具备法庭应用能力？基于前沿模型的印度法律推理评估
Are LLMs Court-Ready? Evaluating Frontier Models on Indian Legal Reasoning
AI总结：
该研究以印度公开法律考试为评估基准，测试前沿大语言模型的法律推理能力，明确了其可辅助的法律工作范畴，以及仍需人类主导的核心法律领域。
摘要：
大语言模型（LLMs）正逐步进入法律工作流程，但我们目前缺乏针对特定司法辖区的框架来评估其在该领域的基础能力。我们以印度的公开法律考试作为透明的评估代理。我们构建的多年度基准集合了来自全国和各州顶级考试的客观题筛选内容，并在真实考试场景下评估开源及前沿LLMs的表现。为了突破选择题的评估局限，我们还纳入了一项由律师评分的配对盲审研究，针对最高法院“出庭律师（Advocate-on-Record）”考试的长文作答进行评估。据我们所知，这是首个基于考试、针对印度的LLM法庭应用能力衡量标准，且同步公开了数据集与评估协议。我们的研究表明，尽管前沿系统在客观题考试中始终能达到历年合格分数线，且表现通常与近期高分考生区间持平甚至更高，但在长文推理任务中，没有任何模型能超越人类最高分考生。评分者的意见集中指出了三类可靠性失效模式：程序或格式合规性问题、权威依据或引用规范问题，以及符合法庭场景的表述风格与结构问题。这些发现明确了LLMs可发挥辅助作用的领域（核查、跨法条一致性校验、法条与判例检索），以及仍需人类主导的核心领域：特定场景的文书起草与提交、程序与救济策略制定、权威依据与例外情形的协调，以及合乎伦理、可问责的判断。
英文摘要：
Large language models (LLMs) are entering legal workflows, yet we lack a jurisdiction-specific framework to assess their baseline competence therein. We use India's public legal examinations as a transparent proxy. Our multi-year benchmark assembles objective screens from top national and state exams and evaluates open and frontier LLMs under real-world exam conditions. To probe beyond multiple-choice questions, we also include a lawyer-graded, paired-blinded study of long-form answers from the Supreme Court's Advocate-on-Record exam. This is, to our knowledge, the first exam-grounded, India-specific yardstick for LLM court-readiness released with datasets and protocols. Our work shows that while frontier systems consistently clear historical cutoffs and often match or exceed recent top-scorer bands on objective exams, none surpasses the human topper on long-form reasoning. Grader notes converge on three reliability failure modes: procedural or format compliance, authority or citation discipline, and forum-appropriate voice and structure. These findings delineate where LLMs can assist (checks, cross-statute consistency, statute and precedent lookups) and where human leadership remains essential: forum-specific drafting and filing, procedural and relief strategy, reconciling authorities and exceptions, and ethical, accountable judgment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
