arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.09849cs.CLcs.AI
收藏

重要性加权可助力大语言模型自改进

Importance Weighting Can Help Large Language Models Self-Improve

Hong Kong University of Science and Technology(香港科技大学)

Chunyang Jiang, Chi-min Chan, Wei Xue, Qifeng Liu, Yike Guo

AI总结:

针对LLM自改进中自生成数据质量不稳定的问题,提出基于Importance Weighting的DS weight指标近似分布偏移程度,结合自一致性过滤样本,仅用极小验证集即可显著提升LLM推理能力,效果比肩依赖外部奖励模型监督的方法。

摘要:

大语言模型(LLM)已在众多任务和应用中展现出卓越能力。然而,在外部监督下使用高质量数据集微调LLM的成本仍然高得令人望而却步。对此,LLM自改进方法近年来得到了蓬勃发展。LLM自改进的典型范式是在自生成数据上训练LLM,由于数据质量不稳定,部分数据可能存在负面影响,需要被过滤掉。当前研究主要采用基于答案正确性的过滤策略,而本文证明,过滤掉答案正确但分布偏移程度(DSE)较高的样本同样能提升自改进效果。鉴于实际样本分布通常难以获取,我们受Importance Weighting方法启发,提出了一种名为DS weight的新指标来近似DSE。随后,我们将DS weight与self-consistency(自一致性)相结合,对自生成样本进行全面过滤,再对语言模型进行微调。实验表明,仅需极小的验证集(规模不超过训练集的5%)来计算DS weight,我们的方法就能显著提升现有LLM自改进方法的推理能力,其最终性能与依赖预训练奖励模型提供外部监督的方法相当。

英文摘要:

Large language models (LLMs) have shown remarkable capability in numerous tasks and applications. However, fine-tuning LLMs using high-quality datasets under external supervision remains prohibitively expensive. In response, LLM self-improvement approaches have been vibrantly developed recently. The typical paradigm of LLM self-improvement involves training LLM on self-generated data, part of which may be detrimental and should be filtered out due to the unstable data quality. While current works primarily employs filtering strategies based on answer correctness, in this paper, we demonstrate that filtering out correct but with high distribution shift extent (DSE) samples could also benefit the results of self-improvement. Given that the actual sample distribution is usually inaccessible, we propose a new metric called DS weight to approximate DSE, inspired by the Importance Weighting methods. Consequently, we integrate DS weight with self-consistency to comprehensively filter the self-generated samples and fine-tune the language model. Experiments show that with only a tiny valid set (up to 5\% size of the training set) to compute DS weight, our approach can notably promote the reasoning ability of current LLM self-improvement methods. The resulting performance is on par with methods that rely on external supervision from pre-trained reward models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑