面向检索增强生成的更轻量且更鲁棒的评估
Towards Lighter and Robust Evaluation for Retrieval Augmented Generation
AI总结：
针对RAG评估依赖昂贵商业大模型且不透明的问题，提出用小型量化开源大模型构建轻量可解释的连续评分指标，还开发了替代人类判断相关性的新AUC指标。
摘要：
大语言模型正促使我们从生成式视角审视更多自然语言处理任务。同时，它们主要通过RAG（检索增强生成）框架提供了一种获取信息的新方式。尽管自回归模型已取得显著进展，但克服生成答案中的幻觉仍是一个持续存在的问题。标准解决方案是使用GPT-4等商业大语言模型来评估这些算法。然而，这类框架成本高昂且透明度不足。因此，我们开展了一项研究，证明了开源权重模型在评估RAG幻觉方面的价值。我们提出了一种轻量级方法，使用更小的量化大语言模型来提供一种可获取且可解释的指标，该指标针对生成答案的正确性和忠实度给出连续评分。该评分使我们能够质疑决策的可靠性，并探索阈值以开发一种新的AUC指标，作为与人类判断相关性的替代方案。
英文摘要：
Large Language Models are prompting us to view more NLP tasks from a generative perspective. At the same time, they offer a new way of accessing information, mainly through the RAG framework. While there have been notable improvements for the autoregressive models, overcoming hallucination in the generated answers remains a continuous problem. A standard solution is to use commercial LLMs, such as GPT4, to evaluate these algorithms. However, such frameworks are expensive and not very transparent. Therefore, we propose a study which demonstrates the interest of open-weight models for evaluating RAG hallucination. We develop a lightweight approach using smaller, quantized LLMs to provide an accessible and interpretable metric that gives continuous scores for the generated answer with respect to their correctness and faithfulness. This score allows us to question decisions' reliability and explore thresholds to develop a new AUC metric as an alternative to correlation with human judgment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
