arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.03642cs.CLcs.LG
收藏

免费自对齐是否可行?

Is Free Self-Alignment Possible?

University of Wisconsin-Madison(威斯康星大学麦迪逊分校)

Dyah Adila, Changho Shin, Yijing Zhang, Frederic Sala

AI总结:

针对预训练语言模型对齐成本高的问题,提出无需额外训练的AlignEZ方法,通过自生成偏好数据和表示编辑实现高效对齐,提升多任务性能,还可加速DPO等昂贵对齐流程。

摘要:

对齐预训练语言模型(LM)通常需要大规模偏好数据和大量计算资源,对于多目标或多元化对齐而言,这些成本甚至高得令人望而却步。这真的有必要吗?我们能否仅利用模型自身能力、无需额外训练就能实现高效对齐?为解答这一问题,我们提出AlignEZ——一种新颖方法,它借助(1)自生成偏好数据和(2)表示编辑来实现高性价比的高效对齐。通过直接作用于已学习的表示,AlignEZ可独立针对不同行为维度,无需传统对齐方法的额外开销。我们的实验表明,这种低成本流程能提升多种任务的性能:即使以强基线模型为起点,通用对齐性能最高提升19.9%,高难度数学推理任务性能提升1.9%。AlignEZ还能同时让模型对齐多个目标,实现对多个偏好维度的细粒度控制。最后我们证明,即使在真实偏好数据有限的情况下,AlignEZ也能加速成本更高的对齐流程(如DPO)。

英文摘要:

Aligning pretrained language models (LMs) often requires large-scale preference data and substantial computational resources. These costs become even more prohibitive for multi-objective or pluralistic alignment. Is this truly necessary? Can we perform efficient alignment using only internal model capabilities, and without additional training? To answer this question, we propose AlignEZ, a novel approach that leverages (1) self-generated preference data and (2) representation editing to achieve cost-effective, efficient alignment. By operating directly on learned representations, AlignEZ independently targets different behavioral aspects without the overhead of traditional alignment methods. Our experiments reveal that this cost-efficient procedure improves performance across diverse tasks: up to 19.9% on general alignment and 1.9% on challenging mathematical reasoning tasks, even when starting from a strong base model. AlignEZ can also align models to multiple objectives simultaneously, granting fine-grained control over multiple preference axes. Finally, we show that AlignEZ can accelerate more expensive alignment procedures--such as DPO--even under limited availability of ground-truth preference data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑