深入探究参数高效偏好对齐技术的权衡取舍
A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
AI总结：
本文通过300多个实验系统研究数据集、对齐技术和模型三个关键因素对参数高效LLM偏好对齐的影响，揭示一致趋势与意外发现，并提出有效对齐的实践指导方针。
摘要：
大型语言模型首先在数万亿个 token 上进行预训练，然后进行指令微调或对齐到特定偏好。由于所需算力巨大，预训练对大多数研究者而言仍然难以企及，而得益于 LoRA 和 QLoRA 等参数高效方法，微调已变得可以负担。已知对齐对许多因素都很敏感，包括数据的数量和质量、对齐方法以及适配器秩。然而，目前尚未有对这些因素在下游性能上影响的广泛研究。为填补这一空白，我们对三个关键轴上的流行选择进行了深入调查：(i) 对齐数据集（HH-RLHF 和 BeaverTails），(ii) 对齐技术（SFT 和 DPO），以及 (iii) 模型（LLaMA-1、Vicuna-v1.3、Mistral-7b 和 Mistral-7b-Instruct）。我们跨越 300 多个实验的广泛设置揭示了一致的趋势和意外的发现。我们观察到信息更丰富的数据如何有助于偏好对齐、监督微调优于偏好优化的情形，以及对齐到不同偏好如何提升下游任务性能。通过深入分析，我们提出了关键指导方针，以帮助研究者执行更有效的参数高效 LLM 对齐。
英文摘要：
Large language models are first pre-trained on trillions of tokens and then instruction-tuned or aligned to specific preferences. While pre-training remains out of reach for most researchers due to the compute required, fine-tuning has become affordable thanks to parameter-efficient methods such as LoRA and QLoRA. Alignment is known to be sensitive to the many factors involved, including the quantity and quality of data, the alignment method, and the adapter rank. However, there has not yet been an extensive study of their effect on downstream performance. To address this gap, we conduct an in-depth investigation of the impact of popular choices for three crucial axes: (i) the alignment dataset (HH-RLHF and BeaverTails), (ii) the alignment technique (SFT and DPO), and (iii) the model (LLaMA-1, Vicuna-v1.3, Mistral-7b, and Mistral-7b-Instruct). Our extensive setup spanning over 300 experiments reveals consistent trends and unexpected findings. We observe how more informative data helps with preference alignment, cases where supervised fine-tuning outperforms preference optimization, and how aligning to a distinct preference boosts performance on downstream tasks. Through our in-depth analyses, we put forward key guidelines to help researchers perform more effective parameter-efficient LLM alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
