知识化上下文微调：探索和利用事实知识以增强上下文学习
Knowledgeable In-Context Tuning: Exploring and Exploiting Factual Knowledge for In-Context Learning
AI总结：
本文提出知识化上下文微调（KICT）框架，通过预训练注入知识、选择高知识相关示例和基于先验知识校准预测，显著提升了LLMs在少样本上下文学习中的文本分类和问答性能。
摘要：
大型语言模型（LLMs）通过将少量带标签的训练示例作为基于文本的提示进行条件化，实现了上下文学习（ICL），从而无需更新参数即可获得具有竞争力的性能。本文从三个核心方面证明事实知识对ICL的性能至关重要：LLMs中学习到的固有知识、从所选上下文示例中提取的事实知识，以及LLMs在输出生成中的知识偏差。为了在少样本学习场景中释放LLMs的能力，我们提出了一种新颖的知识化上下文微调（KICT）框架，以进一步提高ICL的性能：1）在持续自监督预训练过程中向LLMs注入知识；2）明智地选择具有高知识相关性的ICL示例；3）基于先验知识校准预测结果。我们在多个文本分类和问答任务上，对自回归模型（例如GPT风格的LLMs）评估了所提出的方法。实验结果表明，KICT显著优于强基线，在文本分类和问答任务上分别提高了超过13%和7%。
英文摘要：
Large language models (LLMs) enable in-context learning (ICL) by conditioning on a few labeled training examples as a text-based prompt, eliminating the need for parameter updates and achieving competitive performance. In this paper, we demonstrate that factual knowledge is imperative for the performance of ICL in three core facets: the inherent knowledge learned in LLMs, the factual knowledge derived from the selected in-context examples, and the knowledge biases in LLMs for output generation. To unleash the power of LLMs in few-shot learning scenarios, we introduce a novel Knowledgeable In-Context Tuning (KICT) framework to further improve the performance of ICL: 1) injecting knowledge into LLMs during continual self-supervised pre-training, 2) judiciously selecting the examples for ICL with high knowledge relevance, and 3) calibrating the prediction results based on prior knowledge. We evaluate the proposed approaches on autoregressive models (e.g., GPT-style LLMs) over multiple text classification and question-answering tasks. Experimental results demonstrate that KICT substantially outperforms strong baselines and improves by more than 13% and 7% on text classification and question-answering tasks, respectively.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
