干预还是不干预:通过概率模型混合指导推理时对齐
To Intervene or Not: Guiding Inference-time Alignment with Probabilistic Model Blending
AI总结:
提出BlendIn框架,通过质量感知对齐和按可靠性加权混合模型知识,解决推理时对齐中指导有效性差异大的问题,在困难模型对上实现最高50%的性能提升。
摘要:
LLM的广泛部署使得模型对齐成为必要,以确保新训练的模型能够安全有效地响应用户指令。在不同方法中,推理时对齐通常更便宜,因为它仅在输出生成期间进行干预(即提供指导)。现有提案从某些对齐模型中提取指导,但没有适当评估其可靠性。然而,我们的系统评估显示,指导有效性在不同模型间差异很大;由于无效指导会导致进一步混乱和更多干预,由此产生的过度干预通常表明性能较差。为了使干预更有效且更高效,我们引入了BlendIn,一个推理时对齐框架,从二元决策转向创建整合两个模型知识的混合分布。BlendIn通过执行质量感知对齐并根据可靠性按比例加权每个模型的贡献来稳定推理时对齐。与现有工作相比,它保留了有益的指导,同时降低了不可靠建议的权重。BlendIn为未对齐的指导提供了诊断信号和缓解策略,在困难模型对上实现了一致且高达50%的性能提升。我们的代码可在以下网址获取:this https URL。
英文摘要:
The wide deployment of LLMs has made model alignment necessary to make newly trained models safely and effectively respond to user instructions. Among different methods, inference-time alignment is often cheaper as it intervenes (i.e., offers guidances) only during output generation. Existing proposals apply guidances extracted from certain aligned models without properly assessing their reliability. Nonetheless, our systematic evaluation reveals that guidance effectiveness varies drastically across models; since ineffective guidances lead to further confusion and thus further interventions, the resulting excessive interventions typically indicate poor performance. To make interventions more effective and thus more efficient, we introduce BlendIn, an inference-time alignment framework that shifts from binary decisions to creating hybrid distributions integrating both models' knowledge. BlendIn stabilizes inference-time alignment by performing quality-aware alignment and proportionally weighting each model's contribution based on reliability. Compared with existing works, it preserves beneficial guidance while downweighting unreliable suggestions. BlendIn provides both diagnostic signals and mitigation strategies for misaligned guidance, achieving consistent and up to 50% performance improvement on challenging model pairs. Our code is available at: https://github.com/DecayingSeart/BlendIn.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。