以用户为中心对用于解释魁北克保险合同的检索增强生成系统的评估
A Human-Centric Evaluation of a Retrieval-Augmented Generation System for Explaining Quebec Insurance Contracts
AI总结：
研究针对在线保险销售中消费者面临的‘建议差距’问题，对用于解释魁北克汽车保险合同的检索增强生成系统进行以用户为中心的评估，通过用户研究得出系统被视为‘认知均衡器’，但也有局限性，凸显人在回路框架在高风险消费金融中实施AI的重要性。
AI中文摘要：
随着在线保险销售的兴起，消费者面临着巨大的‘建议差距’，需要在没有专家指导的情况下浏览复杂的法律合同。本文对一个旨在使魁北克汽车保险合同更易理解的先进检索增强生成系统进行了以用户为中心的外部评估。通过对拉瓦尔大学154名参与者的用户研究，我们通过测量系统满意度、认知努力、感知自主性和风险来评估该系统在现实世界中的效用。结果表明该系统被视为‘认知均衡器’，在满意度、信任度和清晰度方面获得高分。关键的是，用户对系统提供的自主感的重视甚至超过知识本身，这种影响在金融知识较低的参与者中最为明显。然而，研究也突出了一些局限性：参与者在高风险或情绪化场景中强烈倾向于人工代理。这项工作凸显了人在回路框架在确保人工智能在高风险消费金融中负责任实施的重要性。
英文摘要：
With the rise of online insurance sales, consumers face a significant \enquote{advice gap}, requiring them to navigate complex legal contracts without expert guidance. This paper presents a human-centric, extrinsic evaluation of a state-of-the-art Retrieval-Augmented Generation system, designed to make Quebec automobile insurance contracts more understandable. Through a user study with 154 participants from Laval University, we assess the agent's real-world utility by measuring system satisfaction, cognitive effort, perceived autonomy, and risk. Our results show the system is perceived as a \enquote{cognitive equalizer}, receiving high ratings for satisfaction, trust, and clarity. Crucially, users value the sense of autonomy the system provides even more than the knowledge itself, with this effect being most pronounced among participants with lower financial literacy, demonstrating how such an agent can directly empower individuals. However, our study also highlights some limitations: participants expressed a strong preference for human agents in high-stakes or emotionally charged scenarios. This work highlights the importance of human-in-the-loop frameworks in ensuring the responsible implementation of AI in high-stakes consumer finance.
