P-Aligner：通过原则性指令合成实现语言模型的预对齐
P-Aligner: Enabling Pre-Alignment of Language Models via Principled Instruction Synthesis
AI总结：
P-Aligner提出轻量级模块，通过原则性指令合成和蒙特卡洛树搜索生成符合人类偏好的指令，实现高效预对齐，在多个基准上显著提升模型表现。
摘要：
大型语言模型（LLMs）在与人类用户交互时，被期望产生安全、有用且诚实的内容，但当它们收到有缺陷的指令（例如缺少上下文、模糊的指示或不恰当的语气）时，常常无法与这些价值观对齐，从而在多个维度上留下大量改进空间。一种成本效益高且影响深远的方法是在模型开始解码之前对指令进行预对齐。现有方法要么依赖于高昂的测试时搜索成本，要么依赖于端到端的模型重写，后者由目标不明确的定制训练语料库驱动。在这项工作中，我们证明高效且有效的偏好对齐目标可以通过P-Aligner实现，这是一个轻量级模块，生成保留原始意图但以更符合人类偏好形式表达的指令。P-Aligner在UltraPrompt上训练，这是一个通过所提出的基于原则的流水线使用蒙特卡洛树搜索合成的新数据集，该系统系统地探索与人类偏好紧密相关的候选指令空间。跨不同方法的实验表明，P-Aligner在各种模型和基准上普遍优于强基线，包括在GPT-4-turbo和Gemma-2-SimPO上分别平均获胜率提升28.35%和8.69%。进一步的分析通过多个视角（包括数据质量、搜索策略、迭代部署和时间开销）验证了其有效性和效率。
英文摘要：
Large Language Models (LLMs) are expected to produce safe, helpful, and honest content during interaction with human users, but they frequently fail to align with such values when given flawed instructions, e.g., missing context, ambiguous directives, or inappropriate tone, leaving substantial room for improvement along multiple dimensions. A cost-effective yet high-impact way is to pre-align instructions before the model begins decoding. Existing approaches either rely on prohibitive test-time search costs or end-to-end model rewrite, which is powered by a customized training corpus with unclear objectives. In this work, we demonstrate that the goal of efficient and effective preference alignment can be achieved by P-Aligner, a lightweight module generating instructions that preserve the original intents while being expressed in a more human-preferred form. P-Aligner is trained on UltraPrompt, a new dataset synthesized via a proposed principle-guided pipeline using Monte-Carlo Tree Search, which systematically explores the space of candidate instructions that are closely tied to human preference. Experiments across different methods show that P-Aligner generally outperforms strong baselines across various models and benchmarks, including average win-rate gains of 28.35% and 8.69% on GPT-4-turbo and Gemma-2-SimPO, respectively. Further analyses validate its effectiveness and efficiency through multiple perspectives, including data quality, search strategies, iterative deployment, and time overhead.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
