从指令到约束：基于自动约束验证的语言模型对齐
From Instructions to Constraints: Language Model Alignment with Automatic Constraint Verification
AI总结：
该研究提出ACT框架，通过自动验证约束满足率生成偏好监督信号，经排序学习让语言模型对齐带约束的用户指令，提升多任务性能且约束遵循能力可迁移。
摘要：
用户对齐对于让通用语言模型（LMs）适配下游任务至关重要，但人类标注往往无法覆盖所有类型的指令，尤其是带有自定义约束的指令。我们观察到用户指令通常包含约束条件。虽然从整条指令的角度评估响应质量成本很高，但高效评估约束满足率是可行的。我们研究了NLP任务中的常见约束，根据其参数类型将其分为三类，并提出了统一框架ACT（Aligning to ConsTraints，约束对齐），用于为带约束的用户对齐自动生成监督信号。具体而言，ACT使用实际中通常易于实现的约束验证器，计算每个响应的约束满足率（CSR）。它为每个提示采样多个响应，并根据其CSR自动收集偏好标签。随后，ACT通过基于排序的学习过程让语言模型适配目标任务。在细粒度实体分类、生成式摘要和时间问答任务上的实验表明，ACT能够增强语言模型遵守不同类别约束的能力，从而提升任务性能。进一步的实验显示，这种约束遵循能力具有可迁移性。
英文摘要：
User alignment is crucial for adapting general-purpose language models (LMs) to downstream tasks, but human annotations are often not available for all types of instructions, especially those with customized constraints. We observe that user instructions typically contain constraints. While assessing response quality in terms of the whole instruction is often costly, efficiently evaluating the satisfaction rate of constraints is feasible. We investigate common constraints in NLP tasks, categorize them into three classes based on the types of their arguments, and propose a unified framework, ACT (Aligning to ConsTraints), to automatically produce supervision signals for user alignment with constraints. Specifically, ACT uses constraint verifiers, which are typically easy to implement in practice, to compute constraint satisfaction rate (CSR) of each response. It samples multiple responses for each prompt and collect preference labels based on their CSR automatically. Subsequently, ACT adapts the LM to the target task through a ranking-based learning process. Experiments on fine-grained entity typing, abstractive summarization, and temporal question answering show that ACT is able to enhance LMs' capability to adhere to different classes of constraints, thereby improving task performance. Further experiments show that the constraint-following capabilities are transferable.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
