对抗性数据库提升基于检索的大型语言模型的成功率
Adversarial Databases Improve Success in Retrieval-based Large Language Models
AI总结：
本研究首次证明对抗性信息数据库（如《圣经》文本和随机单词）能提升基于RAG的开源LLMs在医学多项选择题上的成功率，挑战了传统假设。
摘要：
开源大型语言模型（LLMs）作为微调聊天机器人已展现出巨大潜力，并在推理能力上表现出色，超越了许多现有基准。检索增强生成（RAG）是一种通过利用外部知识数据库来提升LLMs在未经明确训练的任务上表现的技术。大量研究已证明，当使用包含相关背景信息的向量数据集时，RAG能更成功地完成下游任务。该领域内人们一直隐含地假设，如果在此背景下使用对抗性背景信息，基于RAG的方法的成功将不复存在，甚至会对结果产生负面影响。为验证这一假设，我们在肾脏病学医学亚专业领域测试了多个开源LLMs，考察RAG在提升其回答多项选择题（MCQ）成功率方面的能力。与以往研究不同，我们考察了RAG在利用相关和对抗性背景数据库时的效果。我们设置了多个开源LLMs，包括Llama 3、Phi-3、Mixtral 8x7b、Zephyr$β$和Gemma 7B Instruct，在零样本RAG流水线中进行测试。作为对抗性信息来源，我们使用了《圣经》文本和随机单词生成的数据库进行比较。我们的数据显示，当整合相关信息向量数据库时，大多数开源LLMs的多项选择题测试成功率如预期般提升。然而令人惊讶的是，对抗性的《圣经》文本显著提升了许多LLMs的成功率，甚至随机单词文本也提升了一些模型的测试能力。总之，我们的结果首次证明了对抗性信息数据集能够提升基于RAG的LLMs成功率这一反直觉的能力。
英文摘要：
Open-source LLMs have shown great potential as fine-tuned chatbots, and demonstrate robust abilities in reasoning and surpass many existing benchmarks. Retrieval-Augmented Generation (RAG) is a technique for improving the performance of LLMs on tasks that the models weren't explicitly trained on, by leveraging external knowledge databases. Numerous studies have demonstrated the effectiveness of RAG to more successfully accomplish downstream tasks when using vector datasets that consist of relevant background information. It has been implicitly assumed by those in the field that if adversarial background information is utilized in this context, that the success of using a RAG-based approach would be nonexistent or even negatively impact the results. To address this assumption, we tested several open-source LLMs on the ability of RAG to improve their success in answering multiple-choice questions (MCQ) in the medical subspecialty field of Nephrology. Unlike previous studies, we examined the effect of RAG in utilizing both relevant and adversarial background databases. We set up several open-source LLMs, including Llama 3, Phi-3, Mixtral 8x7b, Zephyr$β$, and Gemma 7B Instruct, in a zero-shot RAG pipeline. As adversarial sources of information, text from the Bible and a Random Words generated database were used for comparison. Our data show that most of the open-source LLMs improve their multiple-choice test-taking success as expected when incorporating relevant information vector databases. Surprisingly however, adversarial Bible text significantly improved the success of many LLMs and even random word text improved test taking ability of some of the models. In summary, our results demonstrate for the first time the countertintuitive ability of adversarial information datasets to improve the RAG-based LLM success.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
