面向领域聊天机器人的检索增强生成优化强化学习
Reinforcement Learning for Optimizing RAG for Domain Chatbots
AI总结：
本文提出一种基于强化学习的策略模型，通过动态决定是否检索FAQ上下文来优化RAG聊天机器人的LLM token成本，并在略微提升准确率的同时实现显著成本节省。
摘要：
随着大型语言模型（LLM）的出现，对话助手在领域用例中已变得十分普遍。LLM通过训练获得上下文问答能力，而检索增强生成（RAG）进一步使机器人能够回答领域特定问题。本文描述了一种基于RAG的方法，用于构建一个利用常见问题解答（FAQ）数据回答用户查询的聊天机器人。我们使用infoNCE损失训练了一个内部检索嵌入模型，实验结果表明，该内部模型在检索准确率和域外（OOD）查询检测方面均显著优于知名的通用公共嵌入模型。作为LLM，我们使用基于开放API的付费ChatGPT模型。我们注意到，先前检索到的上下文可用于为特定模式/序列的查询（例如后续查询）生成答案。因此，存在优化LLM token数量和成本的空间。在假设检索模型和LLM固定的情况下，我们使用强化学习（RL）优化LLM token数量。具体而言，我们提出了一个位于RAG外部的基于策略的模型，该模型通过策略动作与RAG流水线交互，并更新策略以优化成本。策略模型可以执行两种动作：获取FAQ上下文或跳过检索。我们使用基于开放API的GPT-4作为奖励模型。然后，我们在多个训练聊天会话上使用策略梯度训练策略模型。作为策略模型，我们实验了公共gpt-2模型和内部BERT模型。通过所提出的基于RL的优化与相似度阈值相结合，我们能够在略微提高准确率的同时实现显著的成本节省。尽管我们展示了FAQ聊天机器人的结果，但所提出的RL方法是通用的，可以在任何现有RAG流水线上进行实验。
英文摘要：
With the advent of Large Language Models (LLM), conversational assistants have become prevalent for domain use cases. LLMs acquire the ability to contextual question answering through training, and Retrieval Augmented Generation (RAG) further enables the bot to answer domain-specific questions. This paper describes a RAG-based approach for building a chatbot that answers user's queries using Frequently Asked Questions (FAQ) data. We train an in-house retrieval embedding model using infoNCE loss, and experimental results demonstrate that the in-house model works significantly better than the well-known general-purpose public embedding model, both in terms of retrieval accuracy and Out-of-Domain (OOD) query detection. As an LLM, we use an open API-based paid ChatGPT model. We noticed that a previously retrieved-context could be used to generate an answer for specific patterns/sequences of queries (e.g., follow-up queries). Hence, there is a scope to optimize the number of LLM tokens and cost. Assuming a fixed retrieval model and an LLM, we optimize the number of LLM tokens using Reinforcement Learning (RL). Specifically, we propose a policy-based model external to the RAG, which interacts with the RAG pipeline through policy actions and updates the policy to optimize the cost. The policy model can perform two actions: to fetch FAQ context or skip retrieval. We use the open API-based GPT-4 as the reward model. We then train a policy model using policy gradient on multiple training chat sessions. As a policy model, we experimented with a public gpt-2 model and an in-house BERT model. With the proposed RL-based optimization combined with similarity threshold, we are able to achieve significant cost savings while getting a slightly improved accuracy. Though we demonstrate results for the FAQ chatbot, the proposed RL approach is generic and can be experimented with any existing RAG pipeline.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
