There has been mounting evidence that pretrained language models fine-tuned on large and diverse supervised datasets can transfer well to a variety of out-of-domain tasks. In this work, we investigate this transfer ability to the legal domain. For that, we participated in the legal case entailment task of COLIEE 2021, in which we use such models with no adaptations to the target domain. Our submissions achieved the highest scores, surpassing the second-best team by more than six percentage points. Our experiments confirm a counter-intuitive result in the new paradigm of pretrained language models: given limited labeled data, models with little or no adaptation to the target task can be more robust to changes in the data distribution than models fine-tuned on it. Code is available at https://github.com/neuralmind-ai/coliee.
NOWJ@COLIEE 2025: A Multi-stage Framework Integrating Embedding Models and Large Language Models for Legal Retrieval and Entailment
VNU University of Engineering and Technology（越南工程大学）；Japan Advanced Institute of Science and Technology（日本先进科学研究院）；Center for Juris-Informatics, ROIS-DS（法律信息中心，ROIS-DS）；Research and Development Center for Large Language Models, NII（大型语言模型研发中心，NII）
NeuralMind；University of Campinas (UNICAMP)（坎皮纳斯大学）
Recent work has shown that language models scaled to billions of parameters, such as GPT-3, perform remarkably well in zero-shot and few-shot scenarios. In this work, we experiment with zero-shot models in the legal case entailment task of the COLIEE 2022 competition. Our experiments show that scaling the number of parameters in a language model improves the F1 score of our previous zero-shot result by more than 6 points, suggesting that stronger zero-shot capability may be a characteristic of larger models, at least for this task. Our 3B-parameter zero-shot model outperforms all models, inclu…
NOWJ@COLIEE 2026: Adaptive Pipelines for Legal Retrieval and Reasoning
VNU University of Engineering and Technology（越南国立工程技术大学）；Center for Juris-Informatics, ROIS-DS College of Engineering & Computer Science, VinUniversity（越南Vin大学工程与计算机科学学院法律信息学中心）
Otto von Guericke University Magdeburg（马格德堡奥托·冯·格里克大学）
Getting an overview over the legal domain has become challenging, especially in a broad, international context. Legal question answering systems have the potential to alleviate this task by automatically retrieving relevant legal texts for a specific statement and checking whether the meaning of the statement can be inferred from the found documents. We investigate a combination of the BM25 scoring method of Elasticsearch with word embeddings trained on English translations of the German and Japanese civil law. For this, we define criteria which select a dynamic number of relevant documents ac…
Indian Institute of Technology, Kanpur（印度理工学院坎普尔分校）；Vellore Institute of Technology（韦洛尔理工学院）
This paper describes our submission to the Competition on Legal Information Extraction/Entailment 2022 (COLIEE-2022) workshop on case law competition for tasks 1 and 2. Task 1 is a legal case retrieval task, which involves reading a new case and extracting supporting cases from the provided case law corpus to support the decision. Task 2 is the legal case entailment task, which involves the identification of a paragraph from existing cases that entails the decision in a relevant case. We employed the neural models Sentence-BERT and Sent2Vec for semantic understanding and the traditional retrie…
Large language models that are capable of zero or few-shot prompting approaches have given rise to the new research area of prompt engineering. Recent advances showed that for example Chain-of-Thought (CoT) prompts can improve arithmetic or common sense tasks significantly. We explore how such approaches fare with legal reasoning tasks and take the COLIEE entailment task based on the Japanese Bar exam for testing zero-shot/few-shot and fine-tuning approaches. Our findings show that while CoT prompting and fine-tuning with explanations approaches show improvements, the best results are produced…
Know Your Limits : On the Faithfulness of LLMs as Solvers and Autoformalizers in Legal Reasoning
UC Santa Cruz（加州大学圣克鲁兹分校）；Univ. Helsinki（赫尔辛基大学）；CodeX, Stanford（斯坦福大学CodeX中心）；Stanford University（斯坦福大学）；Canyon Crest Academy（峡谷峰学院）；Monta Vista High School（蒙塔维斯塔高中）；Los Altos High School（洛斯阿尔托斯高中）
Automating Legal Interpretation with LLMs: Retrieval, Generation, and Evaluation
Wangxuan Institute of Computer Technology, Peking University（北京大学王轩计算机技术研究所）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究所）；Peking University Law School（北京大学法学院）
VLSP 2023 -- LTER: A Summary of the Challenge on Legal Textual Entailment Recognition
Vietnam Judicial Academy（越南司法学院）；Hanoi Law University（河内法律大学）；Hung Yen University of Technology and Education（兴安工业大学）；Ho Chi Minh City University of Science（胡志明市科学大学）；Japan Advanced Institute of Science and Technology（日本先进科学技术学院）；The Institute of Statistical Mathematics（统计数学研究所）；National Institute of Informatics（情报学研究所）
Large pre-trained language models (LMs) have demonstrated remarkable ability as few-shot learners. However, their success hinges largely on scaling model parameters to a degree that makes it challenging to train and serve. In this paper, we propose a new approach, named as EFL, that can turn small LMs into better few-shot learners. The key idea of this approach is to reformulate potential NLP task into an entailment one, and then fine-tune the model with as little as 8 examples. We further demonstrate our proposed method can be: (i) naturally combined with an unsupervised contrastive learning-…
Lawma: The Power of Specialization for Legal Annotation
Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）；Tübingen AI Center（图宾根人工智能中心）；Max Planck Institute for Software Systems（马克斯·普朗克软件系统研究所）；ETH Zurich（苏黎世联邦理工学院）；Max Planck Institute for Research on Collective Goods（马克斯·普朗克集体研究研究所）；Washington University in St. Louis - School of Law（圣路易斯华盛顿大学法学院）；University of Virginia School of Law（弗吉尼亚大学法学院）
Black-Box Analysis: GPTs Across Time in Legal Textual Entailment Task
National Institute of Informatics（国立信息学研究所）；University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute（阿尔伯塔机器智能研究所）；Imperial College London（伦敦帝国理工学院）；Royal Holloway, University of London（伦敦大学皇家霍洛威学院）
Research Institute for Artificial Intelligence “Mihai Drăgănescu”（“米哈伊·德勒格内斯库”人工智能研究所）
EuroVoc is a multilingual thesaurus that was built for organizing the legislative documentary of the European Union institutions. It contains thousands of categories at different levels of specificity and its descriptors are targeted by legal texts in almost thirty languages. In this work we propose a unified framework for EuroVoc classification on 22 languages by fine-tuning modern Transformer-based pretrained language models. We study extensively the performance of our trained models and show that they significantly improve the results obtained by a similar tool - JEX - on the same dataset. …
The effectiveness of the BERT model on multiple linguistic tasks has been well documented. On the other hand, its potentials for narrow and specific domains such as Legal, have not been fully explored. In this paper, we examine how BERT can be used in the Arabic legal domain and try customizing this language model for several downstream tasks using several different domain-relevant training and testing datasets to train BERT from scratch. We introduce the AraLegal-BERT, a bidirectional encoder Transformer-based model that have been thoroughly tested and carefully optimized with the goal to amp…
NyayaMind- A Framework for Transparent Legal Reasoning and Judgment Prediction in the Indian Legal System
IIT Kanpur, India（印度理工学院坎普尔分校）；IIT Kharagpur, India（印度理工学院克勒格布尔分校）；Symbiosis Law School Pune, India（印度共生法学院浦那分校）；Dattam Labs, India（印度Dattam实验室）；University of Birmingham, Dubai, United Arab Emirates（英国伯明翰大学迪拜校区）
DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services
Fudan University（复旦大学）；Huazhong University of Science and Technology（华中科技大学）；New York University Shanghai（上海纽约大学）；Northwest University of Political and Law（西北政法大学）
Carnegie Mellon University（卡内基梅隆大学）；Université de Montréal（蒙特利尔大学）
We analyze the ability of pre-trained language models to transfer knowledge among datasets annotated with different type systems and to generalize beyond the domain and dataset they were trained on. We create a meta task, over multiple datasets focused on the prediction of rhetorical roles. Prediction of the rhetorical role a sentence plays in a case decision is an important and often studied task in AI & Law. Typically, it requires the annotation of a large number of sentences to train a model, which can be time-consuming and expensive. Further, the application of the models is restrained to …
Japan Advanced Institute of Science and Technology (JAIST)（日本先端科学技术大学院大学）；Hung Yen University of Education and Technology (UTEHY)（兴安教育技术大学）
In the context of the Competition on Legal Information Extraction/Entailment (COLIEE), we propose a method comprising the necessary steps for finding relevant documents to a legal question and deciding on textual entailment evidence to provide a correct answer. The proposed method is based on the combination of several lexical and morphological characteristics, to build a language model and a set of features for Machine Learning algorithms. We provide a detailed study on the proposed method performance and failure cases, indicating that it is competitive with state-of-the-art approaches on Leg…
