在ChatGPT答案上运用标签模型可提升法律文本蕴含性能
Employing Label Models on ChatGPT Answers Improves Legal Text Entailment Performance
AI总结：
针对temperature>0时ChatGPT法律文本蕴含答案不稳定的问题，提出用弱监督标签模型整合其临时答案为统一标签，在COLIEE 2022数据集上准确率达76.15%，较原SOTA提升8.26%，还分析了错误类型以指导后续研究。
摘要：
法律文本蕴含的目标是确定法律查询中的断言是否能从一条或多条法律条文提供的信息中逻辑推导得出。大语言模型ChatGPT在包括法律文本蕴含在内的众多自然语言处理任务中表现稳健：当我们设置temperature=0（此时ChatGPT的答案具有确定性）并对模型进行提示时，它在COLIEE 2022数据集上达到70.64%的准确率，优于此前67.89%的SOTA（最优水平）。另一方面，若temperature大于0，ChatGPT的答案不具有确定性，会导致答案不一致、结果波动。我们提出利用标签模型（弱监督技术的核心组件）将ChatGPT生成的临时答案整合为统一标签。通过这种方式，我们将ChatGPT的临时答案视为含噪预测，可通过标签模型进行整合。实验结果表明，该方法能达到76.15%的准确率，较此前的最优基准显著提升8.26%。此外，我们对ChatGPT生成错误答案的实例进行了分析，随后对错误进行了分类，所提供的见解可为未来研究工作的潜在改进方向提供指导。
英文摘要：
The objective of legal text entailment is to ascertain whether the assertions in a legal query logically follow from the information provided in one or multiple legal articles. ChatGPT, a large language model, is robust in many natural language processing tasks, including legal text entailment: when we set the temperature = 0 (the ChatGPT answers are deterministic) and prompt the model, it achieves 70.64% accuracy on COLIEE 2022 dataset, which outperforms the previous SOTA of 67.89%. On the other hand, if the temperature is larger than zero, ChatGPT answers are not deterministic, leading to inconsistent answers and fluctuating results. We propose to leverage label models (a fundamental component of weak supervision techniques) to integrate the provisional answers by ChatGPT into consolidated labels. By that way, we treat ChatGPT provisional answers as noisy predictions which can be consolidated by label models. The experimental results demonstrate that this approach can attain an accuracy of 76.15%, marking a significant improvement of 8.26% over the prior state-of-the-art benchmark. Additionally, we perform an analysis of the instances where ChatGPT produces incorrect answers, then we classify the errors, offering insights that could guide potential enhancements for future research endeavors.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
