Icon$^{2}$：通过内在调控利用自合成偏好数据对齐大型语言模型
Icon$^{2}$: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation
AI总结：
本文提出Icon$^{2}$方法，利用LLM表示空间的内在调控构建自合成偏好数据，通过方向向量过滤指令和双向控制生成响应对，显著提升对齐效果并降低计算成本。
摘要：
大型语言模型（LLMs）需要高质量的偏好数据集以与人类偏好对齐。然而，构建此类数据集的传统方法面临重大挑战：依赖预先收集的指令往往导致与目标模型的分布不匹配，而采样多个随机响应则带来大量的计算开销。在本工作中，我们探索了一种范式转变，利用LLMs表示空间的内在调控来高效且定制化地构建偏好数据集，称为Icon$^{2}$。具体而言，它首先提取逐层方向向量以编码复杂的人类偏好，然后利用这些向量基于其内在一致性过滤自合成的指令。在解码过程中，应用双向内在控制来引导token表示，从而精确生成具有清晰对齐差异的响应对。实验结果表明，在对齐和效率方面均有显著提升。Llama3-8B和Qwen2-7B在AlpacaEval 2.0上平均胜率提升13.89%，在Arena-Hard上提升13.45%，同时计算成本降低高达48.1%。
英文摘要：
Large Language Models (LLMs) require high quality preference datasets to align with human preferences. However, conventional methods for constructing such datasets face significant challenges: reliance on pre-collected instructions often leads to distribution mismatches with target models, while the need for sampling multiple stochastic responses introduces substantial computational overhead. In this work, we explore a paradigm shift by leveraging inherent regulation of LLMs' representation space for efficient and tailored preference dataset construction, named Icon$^{2}$. Specifically, it first extracts layer-wise direction vectors to encode sophisticated human preferences and then uses these vectors to filter self-synthesized instructions based on their inherent consistency. During decoding, bidirectional inherent control is applied to steer token representations, enabling the precise generation of response pairs with clear alignment distinctions. Experimental results demonstrate significant improvements in both alignment and efficiency. Llama3-8B and Qwen2-7B achieve an average win rate improvement of 13.89% on AlpacaEval 2.0 and 13.45% on Arena-Hard, while reducing computational costs by up to 48.1%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
