arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.23271cs.CL
收藏

通过分布优化合成扩展LLM知识边界

Scaling LLM Knowledge Boundaries via Distribution-Optimized Synthesis

Zhejiang University(浙江大学);Ant Group(蚂蚁集团);ZJU-Ant Group Joint Lab of Knowledge Graph(浙江大学-蚂蚁集团知识图谱联合实验室)
Songze Li, Yarong Lan, Zhongpu Bo, Zhaoyang Wang, Zhiqiang Liu, Yuan Yuan, Chengtao Gan, Menghao Qian, Enpei Niu, Xiaoke Guo, Yuanxiang Liu, Zhaoyan Gong, Xiang… 展开作者

Songze Li, Yarong Lan, Zhongpu Bo, Zhaoyang Wang, Zhiqiang Liu, Yuan Yuan, Chengtao Gan, Menghao Qian, Enpei Niu, Xiaoke Guo, Yuanxiang Liu, Zhaoyan Gong, Xiangjin Hu, Liangyurui Liu, Jingdian Lu, Lei Liang, Jun Zhou, Huajun Chen, Wen Zhang

AI总结:

提出KDoS框架,通过知识密度驱动的三阶段反馈机制优化合成数据的知识分布,显著扩展LLM知识边界,并在多模型和多数据规模上验证了最优分布的稳定性和有效性。

摘要:

通过合成数据注入知识对于增强大型语言模型(LLM)至关重要。然而,当前的合成方法仅在预设的token数量或固定数据比例处停止,缺乏对知识分布的感知。这导致某些领域稀疏而其他领域冗余,限制了LLM的知识边界。我们从分布角度重新审视知识注入,并假设存在一个最优的知识分布以最大化知识边界扩展。我们提出KDoS(知识分布优化合成),一个通过三阶段反馈机制引入知识密度驱动合成的框架,从盲目生成转向分布优化合成。我们构建了基于维基百科的具有不同知识分布的合成数据,并在0.6B到16B的模型(Qwen、Ling、LLaMA)以及1B到5B token的数据规模上进行实验。我们的主要发现是:(1)最优知识分布始终最大化边界扩展;(2)该分布在不同的骨干网络和规模上保持稳定;(3)KDoS在六个知识基准上优于基线。我们的工作为合成数据驱动的知识注入提供了新的视角和实用框架。

英文摘要:

Knowledge injection via synthetic data is crucial for enhancing Large Language Models (LLMs). However, current synthesis methods simply stop at preset token counts or fixed data ratios, lacking awareness of knowledge distribution. This results in some domains being sparse while others are redundant, limiting LLM knowledge boundaries. We revisit knowledge injection from a distribution perspective and hypothesize that an optimal knowledge distribution exists to maximize knowledge boundary expansion. We propose KDoS (Knowledge Distribution-optimized Synthesis), a framework that introduces knowledge density to drive synthesis through a three-stage feedback mechanism, shifting from blind generation to distribution-optimized synthesis. We construct Wikipedia-based synthetic data with varying knowledge distributions and conduct experiments on models from 0.6B to 16B (Qwen, Ling, LLaMA) and data scales from 1B to 5B tokens. Our key findings are: (1) an optimal knowledge distribution consistently maximizes boundary expansion; (2) this distribution is stable across backbones and scales; (3) KDoS outperforms baselines across six knowledge benchmarks. Our work offers a new perspective and practical framework for synthetic data-driven knowledge injection.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑