Reformatted Alignment（重格式化对齐）
Reformatted Alignment
AI总结：
针对大语言模型对齐微调数据质量提升成本高、易有幻觉错误的问题，本文提出ReAlign方法，通过重格式化指令数据回复提升对齐效果，在少数据、无额外训练技术的情况下显著提升模型多项能力。
摘要：
微调数据的质量对于让大语言模型（LLM）与人类价值观对齐至关重要。当前提升数据质量的方法要么耗费大量人力，要么容易因大语言模型的幻觉产生事实性错误。本文探索提升现有指令数据的质量以更好地匹配人类价值观，提出了一种名为ReAlign的简单有效方法，该方法将指令数据的回复重格式化为更符合预设标准与整理后证据的格式。该方法最大程度减少了人工标注、幻觉问题与规模化扩展的难度，且与现有对齐技术相互兼容。实验表明，ReAlign显著提升了大语言模型的通用对齐能力、数学推理能力、事实准确性与可读性。令人鼓舞的是，在不引入任何额外数据或先进训练技术的情况下，仅通过重格式化回复，LLaMA-2-13B在GSM8K上的数学推理准确率可从46.77%提升至56.63%。此外，仅使用5%的ReAlign数据，就能让基于Alpaca数据集衡量的通用对齐能力提升67%。这项工作凸显了进一步研究大语言模型的科学原理与机制可解释性的必要性。我们已公开相关代码与数据以支持未来研究，访问地址为https://github.com/GAIR-NLP/ReAlign。
英文摘要：
The quality of finetuning data is crucial for aligning large language models (LLMs) with human values. Current methods to improve data quality are either labor-intensive or prone to factual errors caused by LLM hallucinations. This paper explores elevating the quality of existing instruction data to better align with human values, introducing a simple and effective approach named ReAlign, which reformats the responses of instruction data into a format that better aligns with pre-established criteria and the collated evidence. This approach minimizes human annotation, hallucination, and the difficulty in scaling, remaining orthogonal to existing alignment techniques. Experimentally, ReAlign significantly boosts the general alignment ability, math reasoning, factuality, and readability of the LLMs. Encouragingly, without introducing any additional data or advanced training techniques, and merely by reformatting the response, LLaMA-2-13B's mathematical reasoning ability on GSM8K can be improved from 46.77% to 56.63% in accuracy. Additionally, a mere 5% of ReAlign data yields a 67% boost in general alignment ability measured by the Alpaca dataset. This work highlights the need for further research into the science and mechanistic interpretability of LLMs. We have made the associated code and data publicly accessible to support future studies at https://github.com/GAIR-NLP/ReAlign.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
