基于RAG的问答中LLM性能比较：计算机科学文献案例研究
Comparing the Performance of LLMs in RAG-based Question-Answering: A Case Study in Computer Science Literature
AI总结：
本研究比较了四个开源LLM与GPT-3.5在RAG支持下于计算机科学文献问答任务中的表现，发现GPT-3.5和Mistral-7b-instruct表现最佳，同时分析了各模型在准确率、排名和延迟上的差异，证明开源LLM可与专有模型竞争。
摘要：
检索增强生成（Retrieval Augmented Generation，RAG）正逐渐成为一种通过减少幻觉来增强生成式AI模型能力的强大技术。因此，RAG与大型语言模型（LLMs）日益突出的地位引发了人们对不同LLM在不同领域问答（QA）中性能比较的兴趣。本研究比较了四个开源LLM——Mistral-7b-instruct、LLaMa2-7b-chat、Falcon-7b-instruct和Orca-mini-v3-7b，以及OpenAI的热门模型GPT-3.5，在利用RAG支持下的计算机科学文献问答任务上的表现。研究中采用的评估指标包括：二元问题的准确率和精确率、人类专家排名、Google AI模型Gemini排名，以及长答案问题的余弦相似度。GPT-3.5与RAG结合时，能够有效回答二元问题和长答案问题，再次印证了其作为先进LLM的地位。在开源LLM方面，Mistral AI的Mistral-7b-instruct与RAG结合后在回答二元问题和长答案问题上均超越其余模型。然而，在开源LLM中，Orca-mini-v3-7b报告了生成响应的最短平均延迟，而Meta的LLaMa2-7b-chat报告了最高的平均延迟。本研究强调了一个事实：开源LLM同样可以在更好的基础设施支持下与GPT-3.5等专有模型并驾齐驱。
英文摘要：
Retrieval Augmented Generation (RAG) is emerging as a powerful technique to enhance the capabilities of Generative AI models by reducing hallucination. Thus, the increasing prominence of RAG alongside Large Language Models (LLMs) has sparked interest in comparing the performance of different LLMs in question-answering (QA) in diverse domains. This study compares the performance of four open-source LLMs, Mistral-7b-instruct, LLaMa2-7b-chat, Falcon-7b-instruct and Orca-mini-v3-7b, and OpenAI's trending GPT-3.5 over QA tasks within the computer science literature leveraging RAG support. Evaluation metrics employed in the study include accuracy and precision for binary questions and ranking by a human expert, ranking by Google's AI model Gemini, alongside cosine similarity for long-answer questions. GPT-3.5, when paired with RAG, effectively answers binary and long-answer questions, reaffirming its status as an advanced LLM. Regarding open-source LLMs, Mistral AI's Mistral-7b-instruct paired with RAG surpasses the rest in answering both binary and long-answer questions. However, among the open-source LLMs, Orca-mini-v3-7b reports the shortest average latency in generating responses, whereas LLaMa2-7b-chat by Meta reports the highest average latency. This research underscores the fact that open-source LLMs, too, can go hand in hand with proprietary models like GPT-3.5 with better infrastructure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
