基于朴素贝叶斯的大语言模型上下文扩展
Naive Bayes-based Context Extension for Large Language Models
AI总结：
本文提出 NBCE 框架，通过朴素贝叶斯和投票机制扩展大语言模型上下文窗口，无需微调即可提升上下文学习性能。
摘要：
大型语言模型（LLMs）展现出有前景的上下文学习能力。然而，传统的上下文学习（ICL）方法常受限于 transformer 架构的长度限制，在尝试有效整合大量示例的监督信息时面临挑战。本文提出了一种名为 Naive Bayes-based Context Extension (NBCE) 的新框架，通过显著扩展上下文规模，使现有 LLMs 能够利用更多示例执行 ICL。重要的是，这种扩展无需微调或依赖特定模型架构，同时保持线性效率。NBCE 首先将上下文分割为适应目标 LLM 最大长度的等大窗口。接着，引入投票机制选择最相关的窗口作为后验上下文。最后，利用贝叶斯定理生成测试任务。实验结果表明，NBCE 显著提升了性能，尤其在示例数量增加时，始终优于其他替代方法。NBCE 代码将公开。代码地址：https://github.com/amurtadha/NBCE-master
英文摘要：
Large Language Models (LLMs) have shown promising in-context learning abilities. However, conventional In-Context Learning (ICL) approaches are often impeded by length limitations of transformer architecture, which pose challenges when attempting to effectively integrate supervision from a substantial number of demonstration examples. In this paper, we introduce a novel framework, called Naive Bayes-based Context Extension (NBCE), to enable existing LLMs to perform ICL with an increased number of demonstrations by significantly expanding their context size. Importantly, this expansion does not require fine-tuning or dependence on particular model architectures, all the while preserving linear efficiency. NBCE initially splits the context into equal-sized windows fitting the target LLM's maximum length. Then, it introduces a voting mechanism to select the most relevant window, regarded as the posterior context. Finally, it employs Bayes' theorem to generate the test task. Our experimental results demonstrate that NBCE substantially enhances performance, particularly as the number of demonstration examples increases, consistently outperforming alternative methods. The NBCE code will be made publicly accessible. The code NBCE is available at: https://github.com/amurtadha/NBCE-master
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
