开发评估大型语言模型韩语法律语言理解的实用基准
Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models
AI总结：
本文提出KBL基准，通过法律知识、推理任务及韩国律师资格考试，在闭卷和RAG设置下全面评估LLMs的韩语法律语言理解能力，揭示现有模型的改进空间。
摘要：
大型语言模型（LLMs）在法律领域表现出卓越性能，GPT-4甚至通过了美国统一律师资格考试。然而，对于非标准化任务及英语以外的语言任务，其效力依然有限。这凸显了在应用前于每个法律体系内对LLMs进行仔细评估的必要性。在此，我们引入KBL，一个用于评估LLMs韩语法律语言理解的基准，包含（1）7项法律知识任务（510个样本），（2）4项法律推理任务（288个样本），以及（3）韩国律师资格考试（4个领域，53项任务，2510个样本）。前两个数据集是与律师密切合作开发的，以认证方式在实际场景中评估LLMs。此外，考虑到法律从业者研究时频繁使用大量法律文件，我们在闭卷设置（仅依赖内部知识）和检索增强生成（RAG）设置（使用韩语法规和判例语料库）下评估了LLMs。结果表明，仍有巨大的改进空间和机会。
英文摘要：
Large language models (LLMs) have demonstrated remarkable performance in the legal domain, with GPT-4 even passing the Uniform Bar Exam in the U.S. However their efficacy remains limited for non-standardized tasks and tasks in languages other than English. This underscores the need for careful evaluation of LLMs within each legal system before application. Here, we introduce KBL, a benchmark for assessing the Korean legal language understanding of LLMs, consisting of (1) 7 legal knowledge tasks (510 examples), (2) 4 legal reasoning tasks (288 examples), and (3) the Korean bar exam (4 domains, 53 tasks, 2,510 examples). First two datasets were developed in close collaboration with lawyers to evaluate LLMs in practical scenarios in a certified manner. Furthermore, considering legal practitioners' frequent use of extensive legal documents for research, we assess LLMs in both a closed book setting, where they rely solely on internal knowledge, and a retrieval-augmented generation (RAG) setting, using a corpus of Korean statutes and precedents. The results indicate substantial room and opportunities for improvement.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
