Cat-DPO：基于类别的安全对齐
Cat-DPO: Category-Adaptive Safety Alignment
AI总结：
Cat-DPO通过将安全对齐转化为类别约束优化问题，为每个有害类别设置独立的适应性安全边际，提升整体帮助性和无害性，减少类别间的安全方差和最佳至最差差距。
摘要：
对齐大语言模型与人类偏好必须在两个竞争目标之间取得平衡：对合法请求作出有帮助的回应和可靠地拒绝有害请求。大多数基于偏好的安全对齐方法将安全简化为一个单一标量，均匀应用于每对偏好。结果是一个在平均上看起来安全但对少数有害类别相对不安全的模型。我们将安全对齐视为一个按类别约束的优化问题，并推导出Cat-DPO，一种直接偏好优化算法，为每个有害类别提供单独的适应性安全边际。当模型在某一类别上仍产生不安全的回应时，该边际会收紧，一旦模型赶上，该边际会放松，因此训练信号跟踪每个类别的当前难度，而不是在单一全局速率下平均。在两个LLM基础架构和六个偏好学习基线中，Cat-DPO提高了整体帮助性和无害性，并压缩了每类安全方差和最佳至最差差距，提供了一种直接的类别细化改进。
英文摘要：
Aligning large language models with human preferences must balance two competing goals: responding helpfully to legitimate requests and reliably refusing harmful ones. Most preference-based safety alignment methods collapse safety into a single scalar that is applied uniformly to every preference pair. The result is a model that looks safe on average but stays relatively unsafe on a minority of harm categories. We cast safety alignment as a per-category constrained optimization problem and derive Cat-DPO, a direct-preference-optimization algorithm with a separate adaptive safety margin for each harm category. The margin tightens when the model still produces unsafe responses on a category and relaxes once the model catches up, so the training signal tracks each category's current difficulty rather than averaging under one global rate. Across two LLM backbones and six preference-learning baselines, Cat-DPO improves aggregate helpfulness and harmlessness and compresses per-category safety variance and the best-to-worst gap, offering a drop-in per-category refinement of direct preference safety alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
