通过指令遵循视角对偏好学习的系统性考察
A Systematic Examination of Preference Learning through the Lens of Instruction-Following
AI总结：
本研究系统考察了偏好数据集属性对LLM指令遵循任务的影响，通过生成48,000个合成提示并结合RS和MCTS方法，发现共享前缀、高低对比度结合及中等难度训练能显著提升模型对齐和泛化性能。
摘要：
偏好学习是一种广泛采用的训练后技术，用于将大型语言模型（LLMs）与人类偏好对齐，并提升特定的下游任务能力。在本研究中，我们系统地调查了偏好数据集的特定属性如何影响LLMs在指令遵循任务中的对齐和下游性能。我们使用一种新颖的合成数据生成管道，生成了48,000个独特的指令遵循提示，这些提示结合了23个可验证的约束条件，从而能够对模型响应进行细粒度和自动化的质量评估。借助我们的合成提示，我们使用两种偏好数据集整理方法——拒绝采样（RS）和蒙特卡洛树搜索（MCTS）——来获取（被选中，被拒绝）响应对。然后，我们进行实验研究以下因素的影响：（1）被选中和被拒绝响应之间共享前缀的存在，（2）被选中和被拒绝响应的对比度和质量，以及（3）训练提示的复杂性。我们的实验表明，由MCTS生成的偏好对中的共享前缀，在具有挑战性的训练配置中提供了边际但一致的改进和更大的稳定性。高对比度偏好对通常优于低对比度对；然而，将两者结合往往能通过平衡多样性和学习效率来获得最佳性能。此外，与过于具有挑战性的提示相比，在中等难度的提示上进行训练能带来更好的跨任务泛化能力，即使在更复杂的评估场景中也是如此。我们的发现为优化指令遵循任务的偏好数据整理提供了可操作的见解，为增强LLM训练和对齐提供了一个可扩展且有效的框架。
英文摘要：
Preference learning is a widely adopted post-training technique that aligns large language models (LLMs) to human preferences and improves specific downstream task capabilities. In this work we systematically investigate how specific attributes of preference datasets affect the alignment and downstream performance of LLMs in instruction-following tasks. We use a novel synthetic data generation pipeline to generate 48,000 unique instruction-following prompts with combinations of 23 verifiable constraints that enable fine-grained and automated quality assessments of model responses. With our synthetic prompts, we use two preference dataset curation methods - rejection sampling (RS) and Monte Carlo Tree Search (MCTS) - to obtain pairs of (chosen, rejected) responses. Then, we perform experiments investigating the effects of (1) the presence of shared prefixes between the chosen and rejected responses, (2) the contrast and quality of the chosen, rejected responses and (3) the complexity of the training prompts. Our experiments reveal that shared prefixes in preference pairs, as generated by MCTS, provide marginal but consistent improvements and greater stability across challenging training configurations. High-contrast preference pairs generally outperform low-contrast pairs; however, combining both often yields the best performance by balancing diversity and learning efficiency. Additionally, training on prompts of moderate difficulty leads to better generalization across tasks, even for more complex evaluation scenarios, compared to overly challenging prompts. Our findings provide actionable insights into optimizing preference data curation for instruction-following tasks, offering a scalable and effective framework for enhancing LLM training and alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
