Large language models (LLMs) have shown strong performance on legal benchmarks, including multiple-choice components of bar exams. However, their capacity for generating open-ended legal reasoning in realistic scenarios remains insufficiently explored. Notably, to our best knowledge, there are no prior studies or datasets addressing this issue in the Japanese context.
This study presents the first dataset designed to evaluate the open-ended legal reasoning performance of LLMs within the Japanese jurisdiction. The dataset is based on the writing component of the Japanese bar examination, which requires examinees to identify multiple legal issues from long narratives and to construct structured legal arguments in free text format. Our key contribution is the manual evaluation of LLMs' generated responses by legal experts, which reveals limitations and challenges in legal reasoning. Moreover, we conducted a manual analysis of hallucinations to characterize when and how the models introduce content not supported by precedent or law.
Our real exam questions, model-generated responses, and expert evaluations reveal the milestones of current LLMs in the Japanese legal domain. Our dataset and relevant resources will be available online.
KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations
Kyoto University(京都大学);The Hong Kong University of Science and Technology(香港科技大学);Xi’an Jiaotong-Liverpool University(西安交通大学利物浦大学);The University of Tokyo(东京大学)
JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation
The Hong Kong Polytechnic University(香港理工大学);Kyoto University(京都大学);The University of Tokyo(东京大学);Hohai University(淮海大学);University of Science and Technology of China(中国科学技术大学);University of Toronto(多伦多大学)
Technical Report: Small Language Model for Japanese Clinical and Medicine
National Cerebral and Cardiovascular Center Hospital(国立循环器病研究中心医院);Graduate School of Medicine and Faculty of Medicine, Kyoto University(京都大学大学院医学研究科及医学部)
MedZERO: Self-Evolving Agents for Open-Ended Medical Reasoning Through Controlled Knowledge Accumulation
The Chinese University of Hong Kong(香港中文大学);Zhongshan Hospital, Fudan University(复旦大学附属中山医院);South China University of Technology(华南理工大学);Chinese Academy of Sciences(中国科学院)
Rethinking Domain Specialization for Open-Ended Scientific Reasoning in Astronomy Language Models
Oak Ridge National Laboratory(橡树岭国家实验室);The Ohio State University(俄亥俄州立大学);QUP/IPNS, High Energy Accelerator Research Organization (KEK)(高能加速器研究机构(KEK)QUP/IPNS);Max-Planck-Institut für Astronomie(马克斯·普朗克天文研究所)
AIn't Nothing But a Survey? Using Large Language Models for Coding German Open-Ended Survey Responses on Survey Motivation
Social Data Science & AI Lab, LMU Munich(社会科学与人工智能实验室,慕尼黑大学);Munich Center for Machine Learning(慕尼黑机器学习中心);University of Maryland, College Park(马里兰大学学院公园分校);GESIS – Leibniz Institute for the Social Sciences(莱比锡社会科学研究机构)