SPA：通过自优先级优化实现大语言模型对齐中的共识
SPA: Achieving Consensus in LLM Alignment via Self-Priority Optimization
AI总结：
本文提出自优先级对齐（SPA）框架，通过无监督生成、自我评估与双标准去噪构建“先可信后有帮助”的偏好对，并以不确定性加权损失微调模型，在多个基准上实现不损害安全性的有帮助性提升。
摘要：
在高风险场景中——例如自残、法律或医疗咨询——大语言模型（LLM）必须既值得信赖又有帮助。然而，这些目标常常相互冲突。我们提出优先级对齐，这是一种新的对齐范式，强制执行严格的“先可信后有帮助”的排序：对有帮助性的优化以首先满足可信度阈值（例如无害性或诚实性）为条件。为了实现这一目标，我们引入了自优先级对齐（SPA）——一个完全无监督的框架，它生成多样化的响应，由模型自身进行自我评估和优化，并应用双标准去噪来消除不一致性并控制方差。基于此，SPA 构建按字典序排列的偏好对，并使用不确定性加权对齐损失对模型进行微调，该损失强调高置信度、高差距的决策。在多个基准上的实验表明，SPA 在不损害安全性的前提下提高了有帮助性，在保持通用能力的同时优于强基线。我们的结果表明，SPA 为关键的大语言模型应用提供了一种可扩展且可解释的对齐策略。
英文摘要：
In high-stakes scenarios-such as self-harm, legal, or medical queries-LLMs must be both trustworthy and helpful. However, these goals often conflict. We propose priority alignment, a new alignment paradigm that enforces a strict "trustworthy-before-helpful" ordering: optimization of helpfulness is conditioned on first meeting trustworthy thresholds (e.g., harmlessness or honesty). To realize this, we introduce Self-Priority Alignment (SPA)-a fully unsupervised framework that generates diverse responses, self-evaluates them and refines them by the model itself, and applies dual-criterion denoising to remove inconsistency and control variance. From this, SPA constructs lexicographically ordered preference pairs and fine-tunes the model using an uncertainty-weighted alignment loss that emphasizes high-confidence, high-gap decisions. Experiments across multiple benchmarks show that SPA improves helpfulness without compromising safety, outperforming strong baselines while preserving general capabilities. Our results demonstrate that SPA provides a scalable and interpretable alignment strategy for critical LLM applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
