ACER：通过检索实现语言模型上下文自动扩展
ACER: Automatic Language Model Context Extension via Retrieval
AI总结：
本文提出ACER框架，通过短上下文模型模拟人类有损检索过程构建自动数据合成流水线，利用自生成数据微调模型，使其在真实长上下文任务中超越通用模型及原始检索阅读流水线。
摘要：
长上下文建模是语言AI用于消化和推理复杂信息片段的关键能力之一。在实践中，长上下文能力通常通过精心设计的上下文扩展阶段被构建到预训练语言模型（LM）中，旨在产生通用的长上下文能力。然而，在我们的初步实验中，我们发现当前开放权重的通用长上下文模型在实际长上下文处理任务中仍有不足。虽然这意味着完全有效的长上下文建模需要特定任务的数据，但其成本可能令人望而却步。在本文中，我们从人类处理大量信息的方式中汲取灵感：一个有损的检索阶段对大量文档进行排序，而读者最终只深入阅读排名靠前的候选文档。我们构建了一个自动数据合成流水线，利用短上下文LM来模拟这一过程。这些短上下文LM通过使用自生成的数据进行进一步微调，从而获得特定任务的长上下文能力。类似于预训练从不完美数据中学习，我们假设并进一步证明，短上下文模型能够在合成数据上进行自举，不仅在长上下文检索增强生成等真实世界任务中优于长上下文通用模型，也超越了用于合成训练数据的检索与阅读流水线。
英文摘要：
Long-context modeling is one of the critical capabilities of language AI for digesting and reasoning over complex information pieces. In practice, long-context capabilities are typically built into a pre-trained language model~(LM) through a carefully designed context extension stage, with the goal of producing generalist long-context capabilities. In our preliminary experiments, however, we discovered that the current open-weight generalist long-context models are still lacking in practical long-context processing tasks. While this means perfectly effective long-context modeling demands task-specific data, the cost can be prohibitive. In this paper, we draw inspiration from how humans process a large body of information: a lossy \textbf{retrieval} stage ranks a large set of documents while the reader ends up reading deeply only the top candidates. We build an \textbf{automatic} data synthesis pipeline that mimics this process using short-context LMs. The short-context LMs are further tuned using these self-generated data to obtain task-specific long-context capabilities. Similar to how pre-training learns from imperfect data, we hypothesize and further demonstrate that the short-context model can bootstrap over the synthetic data, outperforming not only long-context generalist models but also the retrieval and read pipeline used to synthesize the training data in real-world tasks such as long-context retrieval augmented generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
