arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2411.08147cs.CLcs.AI
收藏

大型语言模型能在长上下文推理中自我改进

Large Language Models Can Self-Improve in Long-context Reasoning

The Chinese University of Hong Kong(香港中文大学);Peking University(北京大学);Tencent(腾讯);Tsinghua University(清华大学)

Siheng Li, Cheng Yang, Zesen Cheng, Lemao Liu, Mo Yu, Yujiu Yang, Wai Lam

AI总结:

本文提出名为\ours的自我改进方法,通过多输出采样、MBR评分及微调或偏好优化,提升LLMs长上下文推理能力,无需依赖外部专家或先进模型标注,在Llama-3.1-8B-Instruct上实现4.2点提升。

摘要:

大型语言模型(LLMs)在处理长上下文方面取得了实质性进展,但在长上下文推理中仍存在困难。现有方法通常涉及使用合成数据对LLMs进行微调,这依赖于人类专家或GPT-4等先进模型的标注,从而限制了进一步的发展。为了解决这个问题,我们研究了LLMs在长上下文推理中自我改进的潜力,并提出了\ours,一种专门为此目的设计的方法。这种方法很直接:我们为每个问题采样多个输出,使用最小贝叶斯风险(MBR)对它们进行评分,然后基于这些输出应用监督微调或偏好优化。在几个领先的LLMs上进行的广泛实验证明了\ours的有效性,Llama-3.1-8B-Instruct获得了4.2个百分点的绝对提升。此外,与依赖人类专家或先进模型生成数据的先前方法相比,\ours取得了更优的性能。我们预计这项工作将为长上下文场景中的自我改进技术开辟新途径,这对于LLMs的持续发展至关重要。

英文摘要:

Large language models (LLMs) have achieved substantial progress in processing long contexts but still struggle with long-context reasoning. Existing approaches typically involve fine-tuning LLMs with synthetic data, which depends on annotations from human experts or advanced models like GPT-4, thus restricting further advancements. To address this issue, we investigate the potential for LLMs to self-improve in long-context reasoning and propose \ours, an approach specifically designed for this purpose. This approach is straightforward: we sample multiple outputs for each question, score them with Minimum Bayes Risk, and then apply supervised fine-tuning or preference optimization based on these outputs. Extensive experiments on several leading LLMs demonstrate the effectiveness of \ours, with an absolute improvement of $4.2$ points for Llama-3.1-8B-Instruct. Furthermore, \ours achieves superior performance compared to prior approaches that depend on data produced by human experts or advanced models. We anticipate that this work will open new avenues for self-improvement techniques in long-context scenarios, which are essential for the continual advancement of LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑