评估基于大语言模型的问答系统中的检索组件
Evaluating the Retrieval Component in LLM-Based Question Answering Systems
AI总结：
本文提出一种用于评估RAG聊天机器人中检索组件的直接基线方法，证明其比传统指标更贴合问答系统整体表现，并考虑了大语言模型忽略无关上下文及产生幻觉的特点。
摘要：
利用大语言模型（LLMs）的问答系统（QA）在很大程度上依赖检索组件为其提供领域特定信息，并降低生成不准确回答或幻觉的风险。尽管对检索器的评估可以追溯到信息检索的早期研究，但在基于大语言模型的聊天机器人中评估其性能仍然是一项挑战。本研究提出了一个用于评估基于检索增强生成（RAG）的聊天机器人中检索器的直接基线。我们的研究结果表明，该评估框架能更好地反映检索器的表现，并且与问答系统的整体性能更加一致。尽管精确率、召回率和F1分数等传统指标可能无法完全捕捉大语言模型的能力——因为即使检索器不完美，它们也可能给出准确回答——但我们的方法考虑了大语言模型忽略不相关上下文的优势，以及其回答中可能存在的错误和幻觉。
英文摘要：
Question answering systems (QA) utilizing Large Language Models (LLMs) heavily depend on the retrieval component to provide them with domain-specific information and reduce the risk of generating inaccurate responses or hallucinations. Although the evaluation of retrievers dates back to the early research in Information Retrieval, assessing their performance within LLM-based chatbots remains a challenge. This study proposes a straightforward baseline for evaluating retrievers in Retrieval-Augmented Generation (RAG)-based chatbots. Our findings demonstrate that this evaluation framework provides a better image of how the retriever performs and is more aligned with the overall performance of the QA system. Although conventional metrics such as precision, recall, and F1 score may not fully capture LLMs' capabilities - as they can yield accurate responses despite imperfect retrievers - our method considers LLMs' strengths to ignore irrelevant contexts, as well as potential errors and hallucinations in their responses.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
