自我提升悖论:语言模型能否在无外部支架的情况下引导推理能力?
The Self-Improvement Paradox: Can Language Models Bootstrap Reasoning Capabilities without External Scaffolding?
AI总结:
针对现有LLM自我提升依赖外部监督的问题,本文提出完全自主的合成问答数据生成框架Crescent,通过诱饵提示生成问题、拒绝采样去重、多数投票收集答案,实现零外部监督下数学推理能力提升及更高效的知识蒸馏。
摘要:
自我提升大型语言模型(LLMs)——即通过利用模型自身生成的合成数据对其进行微调来提升性能——是在避免大量监督的同时推进LLMs能力的一种有前景的方法。现有自我提升方法通常依赖外部监督信号,形式包括种子数据和/或第三方模型的辅助。本文提出Crescent——一个以完全自主方式生成高质量合成问答数据的简单却有效的框架。Crescent首先通过诱饵提示引导LLM生成原始问题,然后利用基于拒绝采样的自我去重对这些问题进行多样化处理,最后将问题输入LLM并通过多数投票收集相应答案。我们表明,Crescent揭示了在数学推理任务中零外部监督信号下真正自我提升的潜力;具体而言,Crescent生成的问答对足以(i)在保留LLM通用性能的同时提升其推理能力(尤其在零样本设置中);(ii)比基于种子数据集增强的现有方法更有效地将LLM知识蒸馏到较弱模型中。
英文摘要:
Self-improving large language models (LLMs) -- i.e., to improve the performance of an LLM by fine-tuning it with synthetic data generated by itself -- is a promising way to advance the capabilities of LLMs while avoiding extensive supervision. Existing approaches to self-improvement often rely on external supervision signals in the form of seed data and/or assistance from third-party models. This paper presents Crescent -- a simple yet effective framework for generating high-quality synthetic question-answer data in a fully autonomous manner. Crescent first elicits the LLM to generate raw questions via a bait prompt, then diversifies these questions leveraging a rejection sampling-based self-deduplication, and finally feeds the questions to the LLM and collects the corresponding answers by means of majority voting. We show that Crescent sheds light on the potential of true self-improvement with zero external supervision signals for math reasoning; in particular, Crescent-generated question-answer pairs suffice to (i) improve the reasoning capabilities of an LLM while preserving its general performance (especially in the 0-shot setting); and (ii) distil LLM knowledge to weaker models more effectively than existing methods based on seed-dataset augmentation.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。