面向多任务设置的自监督互信息对齐探索
An Exploration of Self-Supervised Mutual Information Alignment for Multi-Task Settings
AI总结:
本文探索自监督互信息对齐方法 SAMI 在多任务场景中的表现,发现其在 MT-Bench 上对 DPO 胜率为 57%,在数学任务上虽弱于 SFT,但与 SFT 结合可在多次尝试下带来额外提升。
摘要:
当前,对于能够将语言模型引导至个体属性与偏好的多元化对齐方法的需求日益增长。其中一种方法——基于互信息的自监督对齐(Self-Supervised Alignment with Mutual Information,SAMI)——利用条件互信息来促进行为偏好与模型响应之间的关联。我们开展了两个实验,以探索 SAMI 在多任务设置中的表现。首先,我们在多任务基准(MT-Bench)上将 SAMI 与直接偏好优化(Direct Preference Optimization,DPO)进行比较,使用一个更强的模型为较弱模型生成跨多种类别的训练数据(人文学科、STEM、信息抽取、编程、数学、推理和角色扮演)。结果表明,一轮 SAMI 对 DPO 的胜率为 57%,且不同任务类别之间的性能差异显著。其次,我们考察了 SAMI 相对于监督微调(supervised fine-tuning,SFT)对数学准确率(GSM-8K)的影响。虽然 SAMI 将零样本性能提升了 1.1%,但 SFT 更为有效,提升了 3.2%。不过,SAMI 表现出有趣的扩展趋势。当给予 10 次尝试时,SAMI 将准确率提升了 3.9%,而 SFT 实现了 10.1% 的提升。将 SAMI 与 SFT 结合可在多次尝试设置下带来额外 1.3% 的提升,尽管单次尝试准确率保持不变。
英文摘要:
There is a growing need for pluralistic alignment methods that can steer language models towards individual attributes and preferences. One such method, Self-Supervised Alignment with Mutual Information (SAMI), uses conditional mutual information to encourage the connection between behavioral preferences and model responses. We conduct two experiments exploring SAMI in multi-task settings. First, we compare SAMI to Direct Preference Optimization (DPO) on a multi-task benchmark (MT-Bench), using a stronger model to generate training data for a weaker one across diverse categories (humanities, STEM, extraction, coding, math, reasoning, and roleplay). Our results indicate that one iteration of SAMI has a 57% win rate against DPO, with significant variation in performance between task categories. Second, we examine SAMI's impact on mathematical accuracy (GSM-8K) relative to supervised fine-tuning (SFT). While SAMI increases zero-shot performance by 1.1%, SFT is more effective with a 3.2% boost. However, SAMI shows interesting scaling trends. When given 10 attempts, SAMI improves accuracy by 3.9%, while SFT achieves a 10.1% increase. Combining SAMI with SFT yields an additional improvement of 1.3% in multi-attempt settings, though single-attempt accuracy remains unchanged.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。