ChatQA 2：在长上下文和检索增强生成（RAG）能力上缩小与专有大语言模型的差距
ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities
AI总结：
本研究提出基于Llama 3.0的128K上下文窗口模型ChatQA 2，通过持续训练扩展上下文窗口并三阶段指令调优，在长上下文任务和RAG基准上优于GPT-4-Turbo等模型，且发现多块RAG方案优于直接长上下文方法，开源相关资源。
摘要：
在本研究中，我们提出ChatQA 2，这是一个基于Llama 3.0、具有128K上下文窗口的模型，旨在缩小开源大语言模型（LLMs）与领先专有模型（如GPT-4-Turbo-2024-04-09）在长上下文理解和检索增强生成（RAG）能力上的差距。这两种能力相互补充，对于LLMs处理无法放入单个提示的大量信息至关重要。我们提出了详细的持续训练方案，将Llama3-70B-base的上下文窗口从8K扩展到128K，并通过三阶段指令调优过程增强模型的指令遵循、RAG性能和长上下文理解能力。我们的结果表明，Llama3-ChatQA-2-70B模型在超过100K token的超长任务上，以及仅使用4K上下文窗口的RAG基准测试中，优于大多数现有最先进模型，包括GPT-4-Turbo-2024-04-09、Qwen2-72B-Instruct和Llama3.1-70B-Instruct，显示出跨不同序列长度的强大长上下文能力。我们进一步使用相同的最先进长上下文LLMs，对直接长上下文和RAG解决方案进行了广泛比较。有趣的是，我们发现当检索更多块时，强大的长上下文LLMs使用RAG的性能会提升。在32K和128K基准测试中，使用大量top-k块的RAG持续优于使用相同最先进长上下文模型（如Llama3-ChatQA-2-70B和Qwen2-72B-Instruct）的直接长上下文解决方案。我们向社区开源了模型权重、训练数据和评估设置：https://chatqa2-project.github.io/
英文摘要：
In this work, we introduce ChatQA 2, an Llama 3.0-based model with a 128K context window, designed to bridge the gap between open-source LLMs and leading proprietary models (e.g., GPT-4-Turbo-2024-04-09) in long context understanding and retrieval-augmented generation (RAG) capabilities. These two capabilities are complementary to each other and essential for LLMs to process large volumes of information that cannot fit into a single prompt. We present a detailed continued training recipe to extend the context window of Llama3-70B-base from 8K to 128K tokens, along with a three-stage instruction tuning process to enhance the model's instruction-following, RAG performance, and long-context understanding capabilities. Our results demonstrate that the Llama3-ChatQA-2-70B model outperforms most existing state-of-the-art models, including GPT-4-Turbo-2024-04-09, Qwen2-72B-Instruct, and Llama3.1-70B-Instruct, on ultra-long tasks beyond 100K tokens, as well as on the RAG benchmark using only a 4K context window, showing the strong long context capability across varying sequence lengths. We further provide extensive comparisons between direct long-context and RAG solutions using the same state-of-the-art long-context LLMs. Interestingly, we find that the performance of strong long-context LLMs using RAG improves when retrieving a larger number of chunks. With a large set of top-k chunks, RAG consistently outperforms direct long-context solution using the same state-of-the-art long-context models (e.g., Llama3-ChatQA-2-70B and Qwen2-72B-Instruct) on both 32K and 128K benchmarks. We open-source the model weights, training data, and the evaluation setup for the for the community: https://chatqa2-project.github.io/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
