对比性弱到强泛化
Contrastive Weak-to-strong Generalization
AI总结：
研究针对弱到强泛化中弱模型输出噪声和偏差问题，利用隐式奖励与对比解码的联系，提出ConG框架，通过对比解码生成高质量样本，实现可靠能力转移等，经实证验证其有效性和通用性，推动了弱到强泛化及通向通用人工智能。
摘要：
弱到强泛化是一种很有前景的扩展大语言模型的范式，通过在对齐的较弱模型的样本上训练更强的模型，无需人工反馈或明确奖励建模。然而，弱模型输出中的噪声和偏差阻碍了其鲁棒性和泛化能力，限制了其实际应用。为应对这一挑战，我们利用通过对数似然比近似显式奖励的隐式奖励，并揭示其与对比解码（CD）的结构等价性，CD是一种能减少大语言模型生成中噪声的解码策略。在此基础上，我们提出对比性弱到强泛化（ConG）框架，该框架在对齐前后的弱模型之间采用对比解码来生成更高质量的样本。这种方法实现了更可靠的能力转移、去噪和增强的鲁棒性，显著减轻了传统弱到强方法的局限性。不同模型家族的实证结果证实了持续的改进，证明了ConG的通用性和有效性。我们的发现凸显了ConG在推进弱到强泛化方面的潜力，并为通向通用人工智能提供了一条有前景的途径。
英文摘要：
Weak-to-strong generalization provides a promising paradigm for scaling large language models (LLMs) by training stronger models on samples from aligned weaker ones, without requiring human feedback or explicit reward modeling. However, its robustness and generalization are hindered by the noise and biases in weak-model outputs, which limit its applicability in practice. To address this challenge, we leverage implicit rewards, which approximate explicit rewards through log-likelihood ratios, and reveal their structural equivalence with Contrastive Decoding (CD), a decoding strategy shown to reduce noise in LLM generation. Building on this connection, we propose Contrastive Weak-to-Strong Generalization (ConG), a framework that employs contrastive decoding between pre- and post-alignment weak models to generate higher-quality samples. This approach enables more reliable capability transfer, denoising, and improved robustness, substantially mitigating the limitations of traditional weak-to-strong methods. Empirical results across different model families confirm consistent improvements, demonstrating the generality and effectiveness of ConG. Taken together, our findings highlight the potential of ConG to advance weak-to-strong generalization and provide a promising pathway toward AGI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
