Transformer-based Approaches for Legal Text Processing
Japan Advanced Institute of Science and Technology（日本先进科学技术学院）；Institute of Statistical Mathematics（统计数学研究所）；University of Engineering and Technology, VNU（越南国立大学工程技术学院）；National Institute of Informatics（日本国立信息学研究所）
Ha-Thanh Nguyen, Minh-Phuong Nguyen, Thi-Hai-Yen Vuong, Minh-Quan Bui, Minh-Chau Nguyen, Tran-Binh Dang, Vu Tran, Le-Minh Nguyen, Ken Satoh
备注：
arXiv admin note: substantial text overlap with arXiv:2106.13405
In this paper, we introduce our approaches using Transformer-based models for different problems of the COLIEE 2021 automatic legal text processing competition. Automated processing of legal documents is a challenging task because of the characteristics of legal documents as well as the limitation of the amount of data. With our detailed experiments, we found that Transformer-based pretrained language models can perform well with automated legal text processing problems with appropriate approaches. We describe in detail the processing steps for each task such as problem formulation, data processing and augmentation, pretraining, finetuning. In addition, we introduce to the community two pretrained models that take advantage of parallel translations in legal domain, NFSP and NMSP. In which, NFSP achieves the state-of-the-art result in Task 5 of the competition. Although the paper focuses on technical reporting, the novelty of its approaches can also be an useful reference in automated legal document processing using Transformer-based models.
University of Michigan（密歇根大学）；University of São Paulo（圣保罗大学）；Tikal Tech（蒂卡尔科技公司）；Latam Datalab Serasa Experian（Serasa Experian拉美数据实验室）
We present and make available pre-trained language models (Phraser, Word2Vec, Doc2Vec, FastText, and BERT) for the Brazilian legal language, a Python package with functions to facilitate their use, and a set of demonstrations/tutorials containing some applications involving them. Given that our material is built upon legal texts coming from several Brazilian courts, this initiative is extremely helpful for the Brazilian legal field, which lacks other open and specific tools and language models. Our main objective is to catalyze the use of natural language processing tools for legal texts analy…
Carnegie Mellon University（卡内基梅隆大学）；Université de Montréal（蒙特利尔大学）
We analyze the ability of pre-trained language models to transfer knowledge among datasets annotated with different type systems and to generalize beyond the domain and dataset they were trained on. We create a meta task, over multiple datasets focused on the prediction of rhetorical roles. Prediction of the rhetorical role a sentence plays in a case decision is an important and often studied task in AI & Law. Typically, it requires the annotation of a large number of sentences to train a model, which can be time-consuming and expensive. Further, the application of the models is restrained to …
Carnegie Mellon University（卡内基梅隆大学）；University of Pittsburgh（匹兹堡大学）
Legal texts routinely use concepts that are difficult to understand. Lawyers elaborate on the meaning of such concepts by, among other things, carefully investigating how have they been used in past. Finding text snippets that mention a particular concept in a useful way is tedious, time-consuming, and, hence, expensive. We assembled a data set of 26,959 sentences, coming from legal case decisions, and labeled them in terms of their usefulness for explaining selected legal concepts. Using the dataset we study the effectiveness of transformer-based models pre-trained on large language corpora t…
Legal Prompt Engineering (LPE) or Legal Prompting is a process to guide and assist a large language model (LLM) with performing a natural legal language processing (NLLP) skill. Our goal is to use LPE with LLMs over long legal documents for the Legal Judgement Prediction (LJP) task. We investigate the performance of zero-shot LPE for given facts in case-texts from the European Court of Human Rights (in English) and the Federal Supreme Court of Switzerland (in German, French and Italian). Our results show that zero-shot LPE is better compared to the baselines, but it still falls short compared …
The effectiveness of the BERT model on multiple linguistic tasks has been well documented. On the other hand, its potentials for narrow and specific domains such as Legal, have not been fully explored. In this paper, we examine how BERT can be used in the Arabic legal domain and try customizing this language model for several downstream tasks using several different domain-relevant training and testing datasets to train BERT from scratch. We introduce the AraLegal-BERT, a bidirectional encoder Transformer-based model that have been thoroughly tested and carefully optimized with the goal to amp…
Research Institute for Artificial Intelligence “Mihai Drăgănescu”（“米哈伊·德勒格内斯库”人工智能研究所）
EuroVoc is a multilingual thesaurus that was built for organizing the legislative documentary of the European Union institutions. It contains thousands of categories at different levels of specificity and its descriptors are targeted by legal texts in almost thirty languages. In this work we propose a unified framework for EuroVoc classification on 22 languages by fine-tuning modern Transformer-based pretrained language models. We study extensively the performance of our trained models and show that they significantly improve the results obtained by a similar tool - JEX - on the same dataset. …
Legal artificial intelligence (LegalAI) aims to benefit legal systems with the technology of artificial intelligence, especially natural language processing (NLP). Recently, inspired by the success of pre-trained language models (PLMs) in the generic domain, many LegalAI researchers devote their effort to apply PLMs to legal tasks. However, utilizing PLMs to address legal tasks is still challenging, as the legal documents usually consist of thousands of tokens, which is far longer than the length that mainstream PLMs can process. In this paper, we release the Longformer-based pre-trained langu…
Legal Judgment Reimagined: PredEx and the Rise of Intelligent AI Interpretation in Indian Courts
Symbiosis Law School（symbiosis 法学院）；IISER Kolkata（印度科学教育与研究学院加尔各答分校）；Indian Institute of Technology Kanpur（坎普尔印度理工学院）；Manipal University Jaipur（斋浦尔马尼帕尔大学）
LawGPT: Knowledge-Guided Data Generation and Its Application to Legal LLM
National Key Laboratory for Novel Software Technology（现代软件工程国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
DeliLaw: A Chinese Legal Counselling System Based on a Large Language Model
University of Chinese Academy of Sciences（中国科学院大学）；Shenzhen Institute of Advanced Technology, CAS（中国科学院深圳先进技术研究院）；Hebei University（河北大学）；University of Science and Technology of China（中国科学技术大学）；South University of Science and Technology of China（南方科技大学）
NOWJ@COLIEE 2026: Adaptive Pipelines for Legal Retrieval and Reasoning
VNU University of Engineering and Technology（越南国立工程技术大学）；Center for Juris-Informatics, ROIS-DS College of Engineering & Computer Science, VinUniversity（越南Vin大学工程与计算机科学学院法律信息学中心）
SaulLM-7B: A pioneering Large Language Model for Law
CentraleSupélec（中央高等电力学院）；Université Paris-Saclay（巴黎萨克雷大学）；Sorbonne Université（索邦大学）；Instituto Superior Técnico（里斯本高等理工学院）；Universidade de Lisboa（里斯本大学）；NOVA School of Law（NOVA法学院）
Japan Advanced Institute of Science and Technology（日本北陆先端科学技术大学院大学）
In recent years, thanks to breakthroughs in neural network techniques especially attentive deep learning models, natural language processing has made many impressive achievements. However, automated legal word processing is still a difficult branch of natural language processing. Legal sentences are often long and contain complicated legal terminologies. Hence, models that work well on general documents still face challenges in dealing with legal documents. We have verified the existence of this problem with our experiments in this work. In this dissertation, we selectively present the main ac…
ITMO University（ITMO大学）；Vienna University of Economics and Business (WU)（维也纳经济大学（WU））
Large multi-label text classification is a challenging Natural Language Processing (NLP) problem that is concerned with text classification for datasets with thousands of labels. We tackle this problem in the legal domain, where datasets, such as JRC-Acquis and EURLEX57K labeled with the EuroVoc vocabulary were created within the legal information systems of the European Union. The EuroVoc taxonomy includes around 7000 concepts. In this work, we study the performance of various recent transformer-based models in combination with strategies such as generative pretraining, gradual unfreezing and…
Athens University of Economics and Business（雅典经济与商业大学）；University of Copenhagen（哥本哈根大学）；Cognitiv+
Pre-trained Transformers currently dominate most NLP tasks. They impose, however, limits on the maximum input length (512 sub-words in BERT), which are too restrictive in the legal domain. Even sparse-attention models, such as Longformer and BigBird, which increase the maximum input length to 4,096 sub-words, severely truncate texts in three of the six datasets of LexGLUE. Simpler linear classifiers with TF-IDF features can handle texts of any length, require far less resources to train and deploy, but are usually outperformed by pre-trained Transformers. We explore two directions to cope with…
Deep learning-based Natural Language Processing methods, especially transformers, have achieved impressive performance in the last few years. Applying those state-of-the-art NLP methods to legal activities to automate or simplify some simple work is of great value. This work investigates the value of domain adaptive pre-training and language adapters in legal NLP tasks. By comparing the performance of language models with domain adaptive pre-training on different tasks and different dataset splits, we show that domain adaptive pre-training is only helpful with low-resource downstream tasks, th…
Analysing similarities between legal court documents using natural language processing approaches based on Transformers
TRT5 - Regional Labour Court of the 5th Region（第五地区地区劳工法院）；SENAI CIMATEC University Center（SENAI CIMATEC大学中心）；Surrey Institute for People-Centred AI（萨里以人为本人工智能研究所）；University of Surrey（萨里大学）
University of York（约克大学）；University of Sheffield（谢菲尔德大学）
Natural language processing (NLP) methods for analyzing legal text offer legal scholars and practitioners a range of tools allowing to empirically analyze law on a large scale. However, researchers seem to struggle when it comes to identifying ethical limits to using NLP systems for acquiring genuine insights both about the law and the systems' predictive capacity. In this paper we set out a number of ways in which to think systematically about such issues. We place emphasis on three crucial normative parameters which have, to the best of our knowledge, been underestimated by current debates: …
State Key Laboratory for Novel Software Technology, Software Institute, Nanjing University（南京大学软件学院计算机软件新技术国家重点实验室）；Amazon Alexa AI（亚马逊 Alexa AI）
Given the fact of a case, Legal Judgment Prediction (LJP) involves a series of sub-tasks such as predicting violated law articles, charges and term of penalty. We propose leveraging a unified text-to-text Transformer for LJP, where the dependencies among sub-tasks can be naturally established within the auto-regressive decoder. Compared with previous works, it has three advantages: (1) it fits in the pretraining pattern of masked language models, and thereby can benefit from the semantic prompts of each sub-task rather than treating them as atomic labels, (2) it utilizes a single unified archi…
Japan Advanced Institute of Science and Technology（日本先端科学技术大学院大学）；University of Engineering and Technology, VNU（越南国立大学工程技术大学）；National Institute of Informatics（国立情报学研究所）
COLIEE is an annual competition in automatic computerized legal text processing. Automatic legal document processing is an ambitious goal, and the structure and semantics of the law are often far more complex than everyday language. In this article, we survey and report our methods and experimental results in using deep learning in legal document processing. The results show the difficulties as well as potentials in this family of approaches.
Zero-shot cross-lingual transfer is an important feature in modern NLP models and architectures to support low-resource languages. In this work, We study zero-shot cross-lingual transfer from English to French and German under Multi-Label Text Classification, where we train a classifier using English training set, and we test using French and German test sets. We extend EURLEX57K dataset, the English dataset for topic classification of legal documents, with French and German official translation. We investigate the effect of using some training techniques, namely Gradual Unfreezing and Languag…
In this article, we explore the potential of transformer-based language models (LMs) to correctly represent normative statements in the legal domain, taking tax law as our use case. In our experiment, we use a variety of LMs as bases for both word- and sentence-based clusterers that are then evaluated on a small, expert-compiled test-set, consisting of real-world samples from tax law research literature that can be clearly assigned to one of four normative theories. The results of the experiment show that clusterers based on sentence-BERT-embeddings deliver the most promising results. Based on…
