KoCo:基于知识坐标的语言模型预训练
KoCo: Conditioning Language Model Pre-training on Knowledge Coordinates
AI总结:
KoCo通过将文档映射为三维语义坐标,提升语言模型对现实知识结构的理解,使模型在10个下游任务中性能提升30%并减少幻觉。
摘要:
标准大型语言模型(LLM)预训练通常将语料视为扁平的token序列,忽略了人类依赖的现实上下文。为此,我们引入知识坐标条件(KoCo),一种将每个文档映射为三维语义坐标的简单方法。通过在预训练中将这些坐标作为文本前缀,我们旨在使模型具备显式的上下文意识,以学习文档中的现实知识结构。实验结果表明,KoCo在10个下游任务中显著提升了性能,并使预训练收敛速度加快约30%。此外,我们的分析表明,显式建模知识坐标有助于模型区分稳定事实与噪声,从而有效减轻生成输出中的幻觉。
英文摘要:
Standard Large Language Model (LLM) pre-training typically treats corpora as flattened token sequences, often overlooking the real-world context that humans naturally rely on to contextualize information. To bridge this gap, we introduce Knowledge Coordinate Conditioning (KoCo), a simple method that maps every document into a three-dimensional semantic coordinate. By prepending these coordinates as textual prefixes for pre-training, we aim to equip the model with explicit contextual awareness to learn the documents within the real-world knowledge structure. Experiment results demonstrate that KoCo significantly enhances performance across 10 downstream tasks and accelerates pre-training convergence by approximately 30\%. Furthermore, our analysis indicates that explicitly modeling knowledge coordinates helps the model distinguish stable facts from noise, effectively mitigating hallucination in generated outputs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。