MultiQ&A：通过自动众包问题扰动与答案来衡量鲁棒性的分析
MultiQ&A: An Analysis in Measuring Robustness via Automated Crowdsourcing of Question Perturbations and Answers
AI总结：
本文提出MultiQ&A方法，通过独立LLM智能体大规模众包问题扰动与答案，评估LLM生成回答的鲁棒性和一致性，为机构采用LLM提供衡量置信度与量化幻觉的框架。
摘要：
大型语言模型（LLMs）在机构采用过程中面临的一个关键挑战源于其在生成回答时产生幻觉的倾向。为解决这一问题，我们提出MultiQ&A，这是一种用于评估LLM生成答案的鲁棒性和一致性的系统性方法。我们展示了MultiQ&A通过独立的LLM智能体大规模众包问题扰动及其相应答案的能力。我们的实验最终考察了190万个问题扰动和230万个答案。此外，MultiQ&A表明，集成LLM（如gpt-3.5-turbo）在扰动下仍保持相对鲁棒和一致。MultiQ&A为回答生成空间提供了清晰性，为检查分歧和变异性提供了一种有效方法。因此，我们的系统为机构采用LLM提供了一个潜在框架，具备衡量置信度、一致性以及量化幻觉的能力。
英文摘要：
One critical challenge in the institutional adoption journey of Large Language Models (LLMs) stems from their propensity to hallucinate in generated responses. To address this, we propose MultiQ&A, a systematic approach for evaluating the robustness and consistency of LLM-generated answers. We demonstrate MultiQ&A's ability to crowdsource question perturbations and their respective answers through independent LLM agents at scale. Our experiments culminated in the examination of 1.9 million question perturbations and 2.3 million answers. Furthermore, MultiQ&A shows that ensembled LLMs, such as gpt-3.5-turbo, remain relatively robust and consistent under perturbations. MultiQ&A provides clarity in the response generation space, offering an effective method for inspecting disagreements and variability. Therefore, our system offers a potential framework for institutional LLM adoption with the ability to measure confidence, consistency, and the quantification of hallucinations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
