Finding answers to legal questions about clauses in contracts is an important form of analysis in many legal workflows (e.g., understanding market trends, due diligence, risk mitigation) but more important is being able to do this at scale. Prior work showed that it is possible to use large language models with simple zero-shot prompts to generate structured answers to questions, which can later be incorporated into legal workflows. Such prompts, while effective on simple and straightforward clauses, fail to perform when the clauses are long and contain information not relevant to the question. In this paper, we propose two-stage prompt chaining to produce structured answers to multiple-choice and multiple-select questions and show that they are more effective than simple prompts on more nuanced legal text. We analyze situations where this technique works well and areas where further refinement is needed, especially when the underlying linguistic variations are more than can be captured by simply specifying possible answers. Finally, we discuss future research that seeks to refine this work by improving stage one results by making them more question-specific.
PAKTON: A Multi-Agent Framework for Question Answering in Long Legal Agreements
School of Electrical and Computer Engineering, AILS Laboratory, National Technical University of Athens（电气与计算机工程学院，AILS实验室，国家技术大学雅典）；Instituto de Telecomunicações（电信研究所）
Institute for Theoretical Physics, Utrecht University（乌得勒支大学理论物理研究所）；Center for Complex Systems Studies, Utrecht University（乌得勒支大学复杂系统研究中心）；Utrecht Institute of Linguistics OTS, Utrecht University（乌得勒支大学语言学OTS研究所）
