基于检索增强生成的魁北克汽车保险问答
Quebec Automobile Insurance Question-Answering With Retrieval-Augmented Generation
AI总结：
本文构建了魁北克汽车保险专业知识参考语料库及82个专家问答集，评估了GPT4-o结合RAG架构在保险问答中的表现，发现该专业知识语料库能提升自动与人工评估效果，但指出其仍存在5%至13%的虚假陈述风险，尚不足以在关键领域大规模应用。
摘要：
大型语言模型（LLMs）在各种下游任务中表现优异，且检索增强生成（RAG）架构的使用已被证明能够提升法律问答的性能（Nuruzzaman and Hussain, 2020; Louis et al., 2024）。然而，在保险问答这一特定类型的法律文档领域，其应用仍然有限。本文介绍了两个语料库：魁北克汽车保险专业知识参考语料库，以及一套包含82个外行汽车保险问题的专家解答集。我们的研究利用这两个语料库，通过自动和人工评估方式，对最先进的LLM——GPT4-o回答魁北克汽车保险问题的能力进行了评估。结果表明，平均而言，使用我们的专业知识参考语料库在自动和人工评估指标上均能生成更好的回答。然而，结果也强调，LLM问答在关键领域的大规模应用中仍不够可靠。事实上，我们的结果显示，在已回答的问题中，有5%到13%包含可能导致客户误解的虚假陈述。
英文摘要：
Large Language Models (LLMs) perform outstandingly in various downstream tasks, and the use of the Retrieval-Augmented Generation (RAG) architecture has been shown to improve performance for legal question answering (Nuruzzaman and Hussain, 2020; Louis et al., 2024). However, there are limited applications in insurance questions-answering, a specific type of legal document. This paper introduces two corpora: the Quebec Automobile Insurance Expertise Reference Corpus and a set of 82 Expert Answers to Layperson Automobile Insurance Questions. Our study leverages both corpora to automatically and manually assess a GPT4-o, a state-of-the-art LLM, to answer Quebec automobile insurance questions. Our results demonstrate that, on average, using our expertise reference corpus generates better responses on both automatic and manual evaluation metrics. However, they also highlight that LLM QA is unreliable enough for mass utilization in critical areas. Indeed, our results show that between 5% to 13% of answered questions include a false statement that could lead to customer misunderstanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
