魁北克保险领域大型语言模型基准测试：从闭卷到检索增强生成
Benchmarking Large Language Models for Quebec Insurance: From Closed-Book to Retrieval-Augmented Generation
AI总结：
本文提出AEPC-QA基准测试，评估51个LLM在魁北克保险领域的闭卷和检索增强生成性能，揭示推理、RAG效果及专业化悖论等关键发现。
摘要：
魁北克省保险分销数字化进程加速，特别是在《法案141》等立法变革推动下，已造成显著的“建议缺口”，使消费者在没有专业指导的情况下需自行解读复杂的金融合同。尽管大型语言模型（LLMs）为自动化建议服务提供了可扩展的解决方案，但其在高风险领域中的部署依赖于严格的法律准确性和可信度。本文通过引入AEPC-QA，一个包含807道多选题的私有黄金标准基准测试，该基准测试基于官方监管认证（论文）手册。我们对51个LLM在两种范式下进行了全面评估：闭卷生成和使用专门的魁北克保险文档进行检索增强生成（RAG）。我们的结果揭示了三个关键见解：1）推理时间推理的优越性，其中利用链式思维处理（例如o3-2025-04-16、o1-2024-12-17）的模型显著优于标准指令微调模型；2）RAG作为知识平衡器，使知识薄弱的模型准确率提升超过35个百分点，但悖论性地导致其他模型出现“上下文干扰”，导致性能大幅下降；3）一种“专业化悖论”，即大规模通用模型始终优于较小的、针对法语领域的专用微调模型。这些发现表明，尽管当前架构接近专家水平（约79%），但外部上下文检索引入的不稳定性需要严格鲁棒性校准，才能使自主部署成为可能。
英文摘要：
The digitization of insurance distribution in the Canadian province of Quebec, accelerated by legislative changes such as Bill 141, has created a significant "advice gap", leaving consumers to interpret complex financial contracts without professional guidance. While Large Language Models (LLMs) offer a scalable solution for automated advisory services, their deployment in high-stakes domains hinges on strict legal accuracy and trustworthiness. In this paper, we address this challenge by introducing AEPC-QA, a private gold-standard benchmark of 807 multiple-choice questions derived from official regulatory certification (paper) handbooks. We conduct a comprehensive evaluation of 51 LLMs across two paradigms: closed-book generation and retrieval-augmented generation (RAG) using a specialized corpus of Quebec insurance documents. Our results reveal three critical insights: 1) the supremacy of inference-time reasoning, where models leveraging chain-of-thought processing (e.g. o3-2025-04-16, o1-2024-12-17) significantly outperform standard instruction-tuned models; 2) RAG acts as a knowledge equalizer, boosting the accuracy of models with weak parametric knowledge by over 35 percentage points, yet paradoxically causing "context distraction" in others, leading to catastrophic performance regressions; and 3) a "specialization paradox", where massive generalist models consistently outperform smaller, domain-specific French fine-tuned ones. These findings suggest that while current architectures approach expert-level proficiency (~79%), the instability introduced by external context retrieval necessitates rigorous robustness calibration before autonomous deployment is viable.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
