Transforming legal text into executable decision logic is a longstanding challenge in legal informatics. With the rise of LLMs, this task has gained renewed interest, but remains challenging due to requiring extensive manual coding and evaluation. We use a unique real-world dataset that pairs production-grade decision models with legal text from the Dutch Environment and Planning Act. These models power the Omgevingsloket government platform, where citizens check permit requirements for environmental activities. We study whether intermediate structured representations can improve LLM-based generation of executable decision models from legal text. We compare four input conditions: raw legal text, text enriched with semantic role labels, text enriched with input and output constraints, and text enriched with both. We evaluate along two dimensions: structural evaluation, through similarity to gold decision models with graph kernels and graphs' descriptive statistics, and outcome evaluation, through functional equivalence by executing models on pre-configured test scenarios. Our findings show that I/O constraints provide the dominant improvement (+37-54% similarity over baseline), while semantic role labels show modest improvements. Outcome evaluation shows that generated models match the gold standard on 51-53% of test scenarios, even though generated models are typically smaller and simpler. We find LLMs eliminate redundant pass-through logic that comprises up to 45-55% of nodes. Importantly, structural similarity and outcome equivalence are complementary: structural similarity does not guarantee outcome equivalence, and vice versa. To facilitate reproducibility, we publicly release our dataset of 95 production decision models with associated legal text and all experimental code.
PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice
Alibaba Group（阿里巴巴集团）；Skylenage；Shanghai Jiao Tong University（上海交通大学）；Tsinghua University（清华大学）；University of Waterloo（滑铁卢大学）；University of Pittsburgh（匹兹堡大学）；University of Bologna（博洛尼亚大学）
AILQA: Evaluating AI-Driven Legal Question Answering Systems for the Indian Legal System
Indian Institute of Technology（印度理工学院）；Symbiosis Law School（共生法学院）；Indian Institute of Science Education and Research（印度科学教育与研究学院）；University of Birmingham（伯明翰大学）
LegalAgentBench: Evaluating LLM Agents in Legal Domain
Tsinghua University（清华大学）；Zhipu AI（智谱AI）；Shanghai Amarsoft Enterprise Credit Information Service Co.,Ltd（上海安硕企业征信服务有限公司）；Central South University（中南大学）；University of Waterloo（滑铁卢大学）；University of Notre Dame（圣母大学）
Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyond
College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；Guanghua Law School, Zhejiang University（浙江大学光华法学院）；School of Software Technology, Zhejiang University（浙江大学软件学院）；Law & AI Lab, Zhejiang University（浙江大学法律与人工智能实验室）；Shanghai AI Laboratory（上海人工智能实验室）
DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation
University of Pittsburgh（匹兹堡大学）；Alibaba Group（阿里巴巴集团）；University of Waterloo（滑铁卢大学）；Shandong University（山东大学）；Skylenage；China University of Political Science and Law（中国政法大学）；Qwen Team, Alibaba Group（阿里巴巴集团Qwen团队）；Fordham University（福特汉姆大学）；Shanghai Jiao Tong University（上海交通大学）；Carnegie Mellon University（卡内基梅隆大学）；Tsinghua University（清华大学）
Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study
Department of Data Science & AI, Monash University（数据科学与人工智能系，莫纳什大学）；Faculty of Law, Monash University（法学院，莫纳什大学）
How Do LLMs Understand Legal Philosophy? Diagnosing Legal-Philosophical Reasoning in LLMs
The University of Osaka（大阪大学）；UC Davis（加州大学戴维斯分校）；UC Berkeley（加州大学伯克利分校）；Peking University（北京大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Shanghai Jiao Tong University（上海交通大学）；NII（信息学研究所（日本））；City University of Hong Kong（香港城市大学）
BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law
Technical University of Munich (TUM)（慕尼黑技术大学）；Ludwig Maximilian University of Munich (LMU)（慕尼黑路德维希-马克西米利安大学）；University of Konstanz（康斯坦茨大学）；University of Saarbrücken（萨尔布吕肯大学）
