Ready Jurist One:为动态环境中的法律智能基准测试语言代理
Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments
AI总结:
Ready Jurist One通过动态法律环境和细粒度评估框架,揭示了现有LLM在动态法律任务中的执行缺陷,为法律智能研究提供新方向。
摘要:
静态基准与现实法律实践的动态性之间的差距是推动法律智能发展的关键障碍。为此,我们引入J1-ENVS,首个为基于大语言模型(LLM)的代理设计的交互式和动态法律环境。在法律专家的指导下,它包含来自中国法律实践的六个代表性场景,涵盖三个层次的环境复杂性。我们进一步引入J1-EVAL,一个细粒度的评估框架,旨在在不同法律熟练度水平上评估任务表现和程序合规性。对17个LLM代理的大量实验表明,尽管许多模型显示出扎实的法律知识,但在动态环境中执行程序时仍面临困难。即使最先进的模型GPT-4o,总体表现也未达到60%。这些发现突显了在动态法律智能领域持续存在的挑战,并为未来研究提供了有价值的见解。
英文摘要:
The gap between static benchmarks and the dynamic nature of real-world legal practice poses a key barrier to advancing legal intelligence. To this end, we introduce J1-ENVS, the first interactive and dynamic legal environment tailored for LLM-based agents. Guided by legal experts, it comprises six representative scenarios from Chinese legal practices across three levels of environmental complexity. We further introduce J1-EVAL, a fine-grained evaluation framework, designed to assess both task performance and procedural compliance across varying levels of legal proficiency. Extensive experiments on 17 LLM agents reveal that, while many models demonstrate solid legal knowledge, they struggle with procedural execution in dynamic settings. Even the SOTA model, GPT-4o, falls short of 60% overall performance. These findings highlight persistent challenges in achieving dynamic legal intelligence and offer valuable insights to guide future research.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。