arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2512.02472cs.AIcs.CLcs.LG
收藏

受最小人类监督引导的自演化大语言模型

Guided Self-Evolving LLMs with Minimal Human Supervision

Tencent AI Lab in Seattle(西雅图腾讯AI实验室);Washington University in St. Louis(斯托克维尔华盛顿大学)

Wenhao Yu, Zhenwen Liang, Chengsong Huang, Kishan Panaganti, Tianqing Fang, Haitao Mi, Dong Yu

AI总结:

R-Few通过引入轻量级人类监督,实现稳定可控的自演化大语言模型,提升数学推理性能并减少对大量人类数据的依赖。

摘要:

人工智能的自我进化长期以来被视为实现超智能的途径,其中模型能够自主获取、精炼并内化来自自身学习经验的知识。然而在实践中,无引导的自我进化系统往往很快停滞甚至退化,这种失败源于概念漂移、多样性崩溃和误进化等问题,因为模型会强化自身的偏见并收敛到低熵行为。为了使模型在最小化人类监督的情况下实现稳定且可控的自我进化,我们引入了R-Few,一种受引导的自我博弈挑战者-解决者框架,该框架通过上下文接地和混合训练引入轻量级的人类监督。在每次迭代中,挑战者会采样一小组人工标注的例子来引导合成问题的生成,而解决者则在在线的基于难度的课程中联合训练于人工和合成例子。在数学和一般推理基准测试中,R-F Few实现了持续且迭代的改进。例如,Qwen3-8B-Base在数学任务中比R-Zero提高了3.0分,并在General-Reasoner的性能上实现了相当的水平,尽管后者是基于20倍更多的人类数据进行训练的。消融研究确认了接地挑战者训练和基于课程的解决者训练的互补贡献,进一步分析表明R-Few缓解了漂移,从而产生了更加稳定和可控的共进化动态。

英文摘要:

AI self-evolution has long been envisioned as a path toward superintelligence, where models autonomously acquire, refine, and internalize knowledge from their own learning experiences. Yet in practice, unguided self-evolving systems often plateau quickly or even degrade as training progresses. These failures arise from issues such as concept drift, diversity collapse, and mis-evolution, as models reinforce their own biases and converge toward low-entropy behaviors. To enable models to self-evolve in a stable and controllable manner while minimizing reliance on human supervision, we introduce R-Few, a guided Self-Play Challenger-Solver framework that incorporates lightweight human oversight through in-context grounding and mixed training. At each iteration, the Challenger samples a small set of human-labeled examples to guide synthetic question generation, while the Solver jointly trains on human and synthetic examples under an online, difficulty-based curriculum. Across math and general reasoning benchmarks, R-Few achieves consistent and iterative improvements. For example, Qwen3-8B-Base improves by +3.0 points over R-Zero on math tasks and achieves performance on par with General-Reasoner, despite the latter being trained on 20 times more human data. Ablation studies confirm the complementary contributions of grounded challenger training and curriculum-based solver training, and further analysis shows that R-Few mitigates drift, yielding more stable and controllable co-evolutionary dynamics.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑