arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2412.16686cs.CL
收藏

NILE:大语言模型中的内部一致性对齐

NILE: Internal Consistency Alignment in Large Language Models

The Chinese University of Hong Kong(香港中文大学);City University of Hong Kong(香港城市大学);Huawei Noah’s Ark Lab(华为诺亚实验室);University of Edinburgh(爱丁堡大学)

Minda Hu, Qiyuan Zhang, Yufei Wang, Bowei He, Hongru Wang, Jingyan Zhou, Liangyou Li, Yasheng Wang, Chen Ma, Irwin King

AI总结:

本文提出NILE框架,通过引出并利用LLM内部知识修订指令微调数据集,并结合内部一致性过滤方法筛选样本,显著提升LLM在多个评估基准上的性能。

摘要:

作为增强大语言模型(LLMs)与人类意图对齐的关键步骤,指令微调(IFT)对数据集质量有很高要求。然而,现有IFT数据集往往包含与LLMs在预训练阶段学到的内部知识不一致的内容,这会极大影响IFT的效果。为解决这一问题,我们提出了NILE(iNternal consIstency aLignmEnt,内部一致性对齐)框架,旨在优化IFT数据集以进一步释放LLMs的能力。NILE通过引出目标预训练LLM与指令数据相对应的内部知识来运作,并利用该内部知识修订IFT数据集中的答案。此外,我们提出了一种新颖的内部一致性过滤(ICF)方法,用于筛选训练样本,确保其与LLM内部知识高度一致。实验表明,经NILE对齐的IFT数据集在多个LLM能力评估数据集上显著提升了LLM性能,在Arena-Hard上获得最高66.6%的增益,在Alpaca-Eval V2上获得68.5%的增益。进一步分析证实,NILE框架的每个组成部分都为这些显著的性能提升做出了贡献,并提供了有力证据表明数据集与预训练内部知识的一致性对于最大化LLM潜力至关重要。

英文摘要:

As a crucial step to enhance LLMs alignment with human intentions, Instruction Fine-Tuning (IFT) has a high demand on dataset quality. However, existing IFT datasets often contain knowledge that is inconsistent with LLMs' internal knowledge learned from the pre-training phase, which can greatly affect the efficacy of IFT. To address this issue, we introduce NILE (iNternal consIstency aLignmEnt) framework, aimed at optimizing IFT datasets to unlock LLMs' capability further. NILE operates by eliciting target pre-trained LLM's internal knowledge corresponding to instruction data. The internal knowledge is leveraged to revise the answer in IFT datasets. Additionally, we propose a novel Internal Consistency Filtering (ICF) method to filter training samples, ensuring its high consistency with LLM's internal knowledge. Our experiments demonstrate that NILE-aligned IFT datasets sharply boost LLM performance across multiple LLM ability evaluation datasets, achieving up to 66.6% gain on Arena-Hard and 68.5% on Alpaca-Eval V2. Further analysis confirms that each component of the NILE}framework contributes to these substantial performance improvements, and provides compelling evidence that dataset consistency with pre-trained internal knowledge is pivotal for maximizing LLM potential.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑