We introduce LegalBench-BR, the first public benchmark for evaluating language models on Brazilian legal text classification. The dataset comprises 3,105 appellate proceedings from the Santa Catarina State Court (TJSC), collected via the DataJud API (CNJ) and annotated across five legal areas through LLM-assisted labeling with heuristic validation. On a class-balanced test set, BERTimbau-LoRA, updating only 0.3% of model parameters, achieves 87.6% accuracy and 0.87 macro-F1 (+22pp over Claude 3.5 Haiku, +28pp over GPT-4o mini). The gap is most striking on administrativo (administrative law): GPT-4o mini scores F1 = 0.00 and Claude 3.5 Haiku scores F1 = 0.08 on this class, while the fine-tuned model reaches F1 = 0.91. Both commercial LLMs exhibit a systematic bias toward civel (civil law), absorbing ambiguous classes rather than discriminating them, a failure mode that domain-adapted fine-tuning eliminates. These results demonstrate that general-purpose LLMs cannot substitute for domain-adapted models in Brazilian legal classification, even when the task is a simple 5-class problem, and that LoRA fine-tuning on a consumer GPU closes the gap at zero marginal inference cost. We release the full dataset, model, and pipeline to enable reproducible research in Portuguese legal NLP.
LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
Stanford University（斯坦福大学）；University of Chicago（芝加哥大学）；Maxime Tools（Maxime工具公司）；LawBeta（LawBeta机构）；South Texas College of Law Houston（休斯顿南德克萨斯法学院）；University of Toronto（多伦多大学）；St. Thomas University Benjamin L. Crump College of Law（圣托马斯大学本杰明·L·克伦普法学院）；Harvard Law School（哈佛法学院）；Stanford Center for Legal Informatics - CodeX（斯坦福法律信息学中心（CodeX））；University of Southern California（南加州大学）；Georgetown University Law Center（乔治城大学法律中心）；Stanford Law School（斯坦福法学院）；University of Virginia（弗吉尼亚大学）；Télécom Paris, Institut Polytechnique de Paris（巴黎理工学院巴黎电信学院）
University of Michigan（密歇根大学）；University of São Paulo（圣保罗大学）；Tikal Tech（蒂卡尔科技公司）；Latam Datalab Serasa Experian（Serasa Experian拉美数据实验室）
We present and make available pre-trained language models (Phraser, Word2Vec, Doc2Vec, FastText, and BERT) for the Brazilian legal language, a Python package with functions to facilitate their use, and a set of demonstrations/tutorials containing some applications involving them. Given that our material is built upon legal texts coming from several Brazilian courts, this initiative is extremely helpful for the Brazilian legal field, which lacks other open and specific tools and language models. Our main objective is to catalyze the use of natural language processing tools for legal texts analy…
INACIA: Integrating Large Language Models in Brazilian Audit Courts: Opportunities and Challenges
University of Campinas（坎皮纳斯大学）；National Center for State Courts (NCSC)（国家州法院中心）；Brazilian Association of Jurimetrics (ABJ)（巴西司法计量协会）；Terranova Consultoria（Terranova咨询公司）；Brazilian Federal Court of Accounts (TCU)（巴西联邦审计法院）
In Brazil, all legal professionals must demonstrate their knowledge of the law and its application by passing the OAB exams, the national bar exams. The OAB exams therefore provide an excellent benchmark for the performance of legal information systems since passing the exam would arguably signal that the system has acquired capacity of legal reasoning comparable to that of a human lawyer. This article describes the construction of a new data set and some preliminary experiments on it, treating the problem of finding the justification for the answers to questions. The results provide a baselin…
The Brazilian Supreme Court receives tens of thousands of cases each semester. Court employees spend thousands of hours to execute the initial analysis and classification of those cases -- which takes effort away from posterior, more complex stages of the case management workflow. In this paper, we explore multimodal classification of documents from Brazil's Supreme Court. We train and evaluate our methods on a novel multimodal dataset of 6,510 lawsuits (339,478 pages) with manual annotation assigning each page to one of six classes. Each lawsuit is an ordered sequence of pages, which are stor…
CDJUR-BR -- A Golden Collection of Legal Document from Brazilian Justice with Fine-Grained Named Entities
Universidade de Fortaleza（福塔莱萨大学）；Instituto Federal de Educação, Ciência e Tecnologia do Ceará（塞阿拉联邦教育、科学与技术学院）；Tribunal de Justiça do Estado do Ceará（塞阿拉州法院）
Analysing similarities between legal court documents using natural language processing approaches based on Transformers
TRT5 - Regional Labour Court of the 5th Region（第五地区地区劳工法院）；SENAI CIMATEC University Center（SENAI CIMATEC大学中心）；Surrey Institute for People-Centred AI（萨里以人为本人工智能研究所）；University of Surrey（萨里大学）
Universidade Federal de Alagoas（阿拉戈斯联邦大学）；Pontificia Universidad Católica de Valparaíso（瓦尔帕莱索天主教大学）；Universidade Federal do Rio Grande do Norte（北里奥格兰德联邦大学）；Procuradoria Geral do Estado de Alagoas（阿拉戈斯州总检察院）
Predicting case outcomes is useful but still an extremely hard task for attorneys and other Law professionals. It is not easy to search case information to extract valuable information as this requires dealing with huge data sets and their complexity. For instance, the complexity of Brazil legal system along with the high litigation rates makes this problem even harder. This paper introduces an approach for predicting Brazilian court decisions which is also able to predict whether the decision will be unanimous. We developed a working prototype which performs 79% of accuracy (F1-score) on a da…
Gender Bias Detection in Court Decisions: A Brazilian Case Study
University of Tübingen（蒂宾根大学）；Universidade Estadual de Campinas (UNICAMP)（坎皮纳斯州立大学）；Faculdade de Direito de Ribeirão Preto, Universidade de São Paulo (USP)（圣保罗大学里贝朗普雷托法学院）；Instituto de Computação, Universidade Estadual de Campinas (UNICAMP)（坎皮纳斯州立大学计算机研究所）
The objective of this paper is to develop predictive models to classify Brazilian legal proceedings in three possible classes of status: (i) archived proceedings, (ii) active proceedings, and (iii) suspended proceedings. This problem's resolution is intended to assist public and private institutions in managing large portfolios of legal proceedings, providing gains in scale and efficiency. In this paper, legal proceedings are made up of sequences of short texts called "motions." We combined several natural language processing (NLP) and machine learning techniques to solve the problem. Although…
Even in highly-developed countries, as many as 15-30\% of the population can only understand texts written using a basic vocabulary. Their understanding of everyday texts is limited, which prevents them from taking an active role in society and making informed decisions regarding healthcare, legal representation, or democratic choice. Lexical simplification is a natural language processing task that aims to make text understandable to everyone by replacing complex vocabulary and expressions with simpler ones, while preserving the original meaning. It has attracted considerable attention in the…
