General and legal domain LLMs have demonstrated strong performance in various tasks of LegalAI. However, the current evaluations of these LLMs in LegalAI are defined by the experts of computer science, lacking consistency with the logic of legal practice, making it difficult to judge their practical capabilities. To address this challenge, we are the first to build the Chinese legal LLMs benchmark LAiW, based on the logic of legal practice. To align with the thinking process of legal experts and legal practice (syllogism), we divide the legal capabilities of LLMs from easy to difficult into three levels: basic information retrieval, legal foundation inference, and complex legal application. Each level contains multiple tasks to ensure a comprehensive evaluation. Through automated evaluation of current general and legal domain LLMs on our benchmark, we indicate that these LLMs may not align with the logic of legal practice. LLMs seem to be able to directly acquire complex legal application capabilities but perform poorly in some basic tasks, which may pose obstacles to their practical application and acceptance by legal experts. To further confirm the complex legal application capabilities of current LLMs in legal application scenarios, we also incorporate human evaluation with legal experts. The results indicate that while LLMs may demonstrate strong performance, they still require reinforcement of legal logic.
DeliLaw: A Chinese Legal Counselling System Based on a Large Language Model
University of Chinese Academy of Sciences(中国科学院大学);Shenzhen Institute of Advanced Technology, CAS(中国科学院深圳先进技术研究院);Hebei University(河北大学);University of Science and Technology of China(中国科学技术大学);South University of Science and Technology of China(南方科技大学)
LegalCiteTrust: Benchmarking Citation Trustworthiness in Chinese Long-Form Legal Research Reports
Faculty of Data Science, City University of Macau(澳门城市大学数据科学学院);Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(中国科学院深圳先进技术研究院);Southern University of Science and Technology(南方科技大学)
LawGPT: A Chinese Legal Knowledge-Enhanced Large Language Model
National Key Laboratory for Novel Software Technology, Nanjing University(南京大学计算机软件新技术国家重点实验室);School of Artifical Intelligence, Nanjing University(南京大学人工智能学院);School of Intelligence Science and Technology, Nanjing University(南京大学智能科学与技术学院)
LawLuo: A Multi-Agent Collaborative Framework for Multi-Round Chinese Legal Consultation
College of Computer and Control Engineering, Northeast Forestry University(东北林业大学计算机与控制工程学院);Faculty of Information and Communication Technology, Universiti Tunku Abdul Rahman(拉曼大学信息与通信技术学院);College of Architecture and Arts Design, Shijiazhuang Tiedao University(石家庄铁道大学建筑与艺术设计学院)
LegalMALR:Multi-Agent Query Understanding and LLM-Based Reranking for Chinese Statute Retrieval
Faculty of Data Science(数据科学学院);City University of Macau(澳门城市大学);Shenzhen Key Laboratory for High Performance Data Mining(深圳高性能数据挖掘重点实验室);Shenzhen Institutes of Advanced Technology(深圳先进技术研究所);Chinese Academy of Sciences(中国科学院);Southern University of Science and Technology(南方科技大学);Artificial Intelligence Research Institute(人工智能研究院);Shenzhen University of Advanced Technology(深圳大学先进技术学院)
Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyond
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Guanghua Law School, Zhejiang University(浙江大学光华法学院);School of Software Technology, Zhejiang University(浙江大学软件学院);Law & AI Lab, Zhejiang University(浙江大学法律与人工智能实验室);Shanghai AI Laboratory(上海人工智能实验室)
JPO: Juris Policy Optimization for Structured Legal Reasoning in Criminal Judgment Prediction
Tencent(腾讯);Peking University(北京大学);Tsinghua University(清华大学);City University of Hong Kong(香港城市大学);University of California(加利福尼亚大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Zhejiang University(浙江大学);University of Hong Kong(香港大学)
DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services
Fudan University(复旦大学);Huazhong University of Science and Technology(华中科技大学);New York University Shanghai(上海纽约大学);Northwest University of Political and Law(西北政法大学)
Legal Syllogism Prompting: Teaching Large Language Models for Legal Judgment Prediction
Peking University Law School(北京大学法学院);Institute for Artificial Intelligence, Peking University(北京大学人工智能研究院);PKU-WUHAN Institute for Artificial Intelligence(北大-武汉人工智能研究院)
An Explicit Syllogistic Legal Reasoning Framework for Large Language Models
Gaoling School of Artificial Intelligence(人工智能学院);Renmin University of China(中国人民大学);School of Information Technology and Management(信息科技与管理学院);University of International Business and Economics(国际经济贸易大学)
LegalOne: A Family of Foundation Models for Reliable Legal Reasoning
Department of Computer Science, Tsinghua University(清华大学计算机科学系);Quancheng Laboratory(清华云城实验室);University of Waterloo, Canada(加拿大滑铁卢大学);China University of Political Science and Law(中国政法大学);MegaTech.AI Inc(MegaTech.AI公司)
Legal Rule Induction: Towards Generalizable Principle Discovery from Analogous Judicial Precedents
Department of Computer Science and Engineering, HKUST, Hong Kong SAR, China(香港科技大学计算机科学与工程系);School of Law, Tsinghua University, Beijing, China(清华大学法学院)
LexRel: Benchmarking Legal Relation Extraction for Chinese Civil Cases
Tsinghua University(清华大学);Peking University(北京大学);University of Glasgow(格拉斯哥大学);Beijing University of Posts and Telecommunications(北京邮电大学);Queen Mary University of London(伦敦玛丽女王大学);Northeastern University(东北大学)
DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation
University of Pittsburgh(匹兹堡大学);Alibaba Group(阿里巴巴集团);University of Waterloo(滑铁卢大学);Shandong University(山东大学);Skylenage;China University of Political Science and Law(中国政法大学);Qwen Team, Alibaba Group(阿里巴巴集团Qwen团队);Fordham University(福特汉姆大学);Shanghai Jiao Tong University(上海交通大学);Carnegie Mellon University(卡内基梅隆大学);Tsinghua University(清华大学)
LegalAgentBench: Evaluating LLM Agents in Legal Domain
Tsinghua University(清华大学);Zhipu AI(智谱AI);Shanghai Amarsoft Enterprise Credit Information Service Co.,Ltd(上海安硕企业征信服务有限公司);Central South University(中南大学);University of Waterloo(滑铁卢大学);University of Notre Dame(圣母大学)
AgentsCourt: Building Judicial Decision-Making Agents with Court Debate Simulation and Legal Knowledge Augmentation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
IPEval: A Bilingual Intellectual Property Agency Consultation Evaluation Benchmark for Large Language Models
School of Future Technology, Dalian University of Technology(大连理工大学未来技术学院);School of Computer Science and Technology, Dalian University of Technology(大连理工大学计算机科学与技术学院);School of Public Administration and Policy, Dalian University of Technology(大连理工大学公共管理与政策学院)
The Cases LJP Never Sees: Prosecution Decision Prediction for More Complete Criminal Liability Assessment
Beijing Institute of Technology(北京理工大学);Osaka University(大阪大学);Xi’an University of Technology(西安理工大学);Institute of Science Tokyo(东京科学研究所);City University of Hong Kong(香港城市大学)
LEVEN: A Large-Scale Chinese Legal Event Detection Dataset
Tsinghua University(清华大学);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);International Innovation Center of Tsinghua University(清华大学国际创新中心);Beijing Academy of Artificial Intelligence(北京智源人工智能研究院);Beijing Powerlaw Intelligent Technology Co., Ltd.(北京幂律智能科技有限责任公司)
Solving the Unsolvable: Translating Case Law in Hong Kong
UOW College Hong Kong(香港UOW学院);City University of Hong Kong(香港城市大学);The Chinese University of Hong Kong Shenzhen(香港中文大学深圳校区);The University of Hong Kong(香港大学)