arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.05620cs.CL
收藏

LAiW:一个中文法律大语言模型基准

LAiW: A Chinese Legal Large Language Models Benchmark

Sichuan University(四川大学);ChanceFocus (Shanghai) AMC(ChanceFocus(上海)资产管理公司);Wuhan University(武汉大学);Southwest Petroleum University(西南石油大学)

Yongfu Dai, Duanyu Feng, Jimin Huang, Haochen Jia, Qianqian Xie, Yifang Zhang, Weiguang Han, Wei Tian, Hao Wang

AI总结:

针对现有法律LLM评估缺乏法律实践逻辑的问题,研究基于法律三段论构建了中文法律LLM基准LAiW,将能力划分为三个层级并发现当前LLM虽具备复杂应用能力但基础任务表现不佳,仍需强化法律逻辑。

摘要:

通用及法律领域的LLM在LegalAI的各项任务中展现了强大的性能。然而,目前LegalAI中对这些LLM的评估由计算机科学专家定义,缺乏与法律实践逻辑的一致性,难以判断其实际能力。为应对这一挑战,我们基于法律实践逻辑,率先构建了中文法律LLM基准LAiW。为契合法律专家和法律实践的思维过程(三段论),我们将LLM的法律能力由易到难划分为三个层级:基础信息检索、法律基础推理和复杂法律应用。每个层级包含多个任务以确保全面评估。通过在我们的基准上对当前通用及法律领域LLM进行自动化评估,我们发现这些LLM可能与法律实践逻辑不一致。LLM似乎能直接获得复杂法律应用能力,但在某些基础任务中表现不佳,这可能对其在实际应用及法律专家的接受度造成阻碍。为进一步确认当前LLM在法律应用场景中的复杂法律应用能力,我们还引入了法律专家的人工评估。结果表明,尽管LLM可能展现出强大的性能,但仍需强化法律逻辑。

英文摘要:

General and legal domain LLMs have demonstrated strong performance in various tasks of LegalAI. However, the current evaluations of these LLMs in LegalAI are defined by the experts of computer science, lacking consistency with the logic of legal practice, making it difficult to judge their practical capabilities. To address this challenge, we are the first to build the Chinese legal LLMs benchmark LAiW, based on the logic of legal practice. To align with the thinking process of legal experts and legal practice (syllogism), we divide the legal capabilities of LLMs from easy to difficult into three levels: basic information retrieval, legal foundation inference, and complex legal application. Each level contains multiple tasks to ensure a comprehensive evaluation. Through automated evaluation of current general and legal domain LLMs on our benchmark, we indicate that these LLMs may not align with the logic of legal practice. LLMs seem to be able to directly acquire complex legal application capabilities but perform poorly in some basic tasks, which may pose obstacles to their practical application and acceptance by legal experts. To further confirm the complex legal application capabilities of current LLMs in legal application scenarios, we also incorporate human evaluation with legal experts. The results indicate that while LLMs may demonstrate strong performance, they still require reinforcement of legal logic.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑