ARES：面向检索增强生成系统的自动化评估框架
ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems
AI总结：
ARES 通过合成训练数据微调轻量级语言模型评判器，并结合少量人工标注进行预测驱动推断，在仅需数百条人工标注的情况下，从上下文相关性、答案忠实度和答案相关性三个维度准确评估 RAG 系统，且能适应领域偏移。
摘要：
评估检索增强生成（RAG）系统传统上依赖人工标注输入查询、待检索段落和待生成响应。我们提出 ARES，一个自动化 RAG 评估系统，用于从上下文相关性、答案忠实度和答案相关性三个维度评估 RAG 系统。ARES 通过自行创建合成训练数据，微调轻量级语言模型评判器，以评估 RAG 各组成部分的质量。为缓解潜在的预测误差，ARES 利用少量人工标注数据点进行预测驱动推断（PPI）。在 KILT、SuperGLUE 和 AIS 中的八个不同知识密集型任务上，ARES 仅使用几百条人工标注即可准确评估 RAG 系统。此外，ARES 评判器在领域偏移下仍保持有效，即使在改变被评估 RAG 系统所使用的查询和/或文档类型后仍能保持准确。我们已在 Github 上公开代码和数据集。
英文摘要：
Evaluating retrieval-augmented generation (RAG) systems traditionally relies on hand annotations for input queries, passages to retrieve, and responses to generate. We introduce ARES, an Automated RAG Evaluation System, for evaluating RAG systems along the dimensions of context relevance, answer faithfulness, and answer relevance. By creating its own synthetic training data, ARES finetunes lightweight LM judges to assess the quality of individual RAG components. To mitigate potential prediction errors, ARES utilizes a small set of human-annotated datapoints for prediction-powered inference (PPI). Across eight different knowledge-intensive tasks in KILT, SuperGLUE, and AIS, ARES accurately evaluates RAG systems while using only a few hundred human annotations during evaluation. Furthermore, ARES judges remain effective across domain shifts, proving accurate even after changing the type of queries and/or documents used in the evaluated RAG systems. We make our code and datasets publicly available on Github.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
