使用大语言模型对车载对话系统进行自动化事实基准测试
Automated Factual Benchmarking for In-Car Conversational Systems using Large Language Models
AI总结：
本文提出一种基于LLM的车载对话系统自动化事实基准测试方法，结合GPT-4与提示技术，在事实正确性上与专家评估达成超90%一致率，且响应高效。
摘要：
车载对话系统有望提升车内用户体验。现代对话系统基于大语言模型（LLMs），这使其容易出现幻觉等错误，即不准确、虚构且因此事实上不正确的信息。本文提出了一种基于LLM的方法，用于对车载对话系统进行自动化事实基准测试。我们用五种基于LLM的方法实例化了该方法，利用集成技术和多样化的人设来提高一致性并最小化幻觉。我们使用该方法评估了CarExpert（一个车载检索增强对话问答系统）相对于车辆手册的事实正确性。我们制作了一个专为车载领域创建的新数据集，并针对专家评估测试了该方法。结果显示，GPT-4与Input Output Prompting的组合在事实正确性上与专家评估达成超过90%的一致率，且是最有效的方法，平均响应时间为4.5秒。研究结果表明，基于LLM的测试是验证对话系统事实正确性的可行方法。
英文摘要：
In-car conversational systems bring the promise to improve the in-vehicle user experience. Modern conversational systems are based on Large Language Models (LLMs), which makes them prone to errors such as hallucinations, i.e., inaccurate, fictitious, and therefore factually incorrect information. In this paper, we present an LLM-based methodology for the automatic factual benchmarking of in-car conversational systems. We instantiate our methodology with five LLM-based methods, leveraging ensembling techniques and diverse personae to enhance agreement and minimize hallucinations. We use our methodology to evaluate CarExpert, an in-car retrieval-augmented conversational question answering system, with respect to the factual correctness to a vehicle's manual. We produced a novel dataset specifically created for the in-car domain, and tested our methodology against an expert evaluation. Our results show that the combination of GPT-4 with the Input Output Prompting achieves over 90 per cent factual correctness agreement rate with expert evaluations, other than being the most efficient approach yielding an average response time of 4.5s. Our findings suggest that LLM-based testing constitutes a viable approach for the validation of conversational systems regarding their factual correctness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
