每个样本都很重要：基于逐点约束的语言模型监督微调
Every Sample Counts: Supervised Fine-Tuning of Language Models with Pointwise Constraints
AI总结：
研究语言模型微调中逐点约束问题，提出新对齐框架，通过逐样本约束最小化平均损失，引入学习松弛方法并开发增广拉格朗日方法，在多任务和约束下实例化，减少约束违反且保留模型性能。
AI中文摘要：
微调语言模型通常需要在不影响下游性能的情况下对单个输入施加约束。现有的约束对齐方法平均施加约束，这可能会在输入或用户之间引发不良差异。我们提出了一种新颖的对齐框架，通过在最小化平均损失的同时施加逐样本约束来解决这一差距。为减轻过度严格的约束和异常值的影响，我们引入了一种基于样本的学习松弛方法，以最小化调整约束，在用户定义的松弛成本和训练目标之间进行权衡。为应对实际的对偶性和优化挑战，我们开发了一种针对此公式的增广拉格朗日方法。我们通过在不同的小语言模型微调任务和约束下实例化该框架，展示了其灵活性：指令遵循中的安全性、函数调用中的偏好以及重排中的长度。在这些设置中，我们的方法减少了尾部约束违反，同时很大程度上保留了模型的性能。
英文摘要：
Fine-tuning language models often requires enforcing constraints on individual inputs without compromising downstream performance. Existing constrained alignment methods impose constraints on average, which can induce undesirable disparities across inputs or users. We propose a novel alignment framework that addresses this gap by enforcing per-sample constraints while still minimizing an average loss. To mitigate the impact of overly restrictive constraints and outliers, we introduce a learned, sample-dependent relaxation that minimally adjusts the constraints, trading off a user-defined relaxation cost with the training objective. To address practical duality and optimization challenges, we develop an augmented Lagrangian approach tailored to this formulation. We demonstrate the flexibility of the framework by instantiating it under distinct small language-model fine-tuning tasks and constraints: safety in instruction following, preferences in function calling and length in re-ranking. Across these settings, our approach reduces tail constraint violations while largely preserving the model's performance.
