面向规范学习：从偏好对进行推理时对齐
Towards Spec Learning: Inference-Time Alignment from Preference Pairs
AI总结：
提出规范学习框架，利用少量用户指令和偏好判断生成自然语言规范，在推理时引导LLM行为，无需参数更新，在密集偏好信号领域优于DPO，且规范可解释。
摘要：
将大型语言模型（LLM）导向期望行为通常依赖于基于对模型响应的仔细检查而手工制作提示的迭代过程。这是一个复杂、脆弱且易出错的过程。基于偏好的微调是一种更严格但通常成本过高的解决方案。我们提出规范学习，一个依赖于简短用户指令和少量偏好判断的框架。这些被编译成自然语言提示形式的规范，用于LLM。规范在推理时条件化LLM，不需要对底层模型进行参数更新。我们表明，基于编译规范生成的响应在偏好信号密集的专业领域数据集上通常优于直接偏好优化（DPO）。与不透明的权重更新不同，生成的规范是人类可读的，并且作为产生它们的偏好信号的可解释和透明的书面体现。
英文摘要：
Steering a large language model (LLM) toward a desired behavior typically relies on an iterative process of hand-crafting a prompt based on a careful inspection of the model's responses. This is an involved, brittle, and error-prone process. Preference-based fine-tuning is a more rigorous but often prohibitively expensive solution. We propose spec learning, a framework that relies on a brief user instruction and a small set of preference judgments. These are compiled into specifications in the form of natural-language prompts for an LLM. Specifications condition LLMs at inference time, and no parameter updates to the underlying models are required. We show that the responses generated based on the compiled specifications often outperform direct preference optimization (DPO) on datasets from specialized domains whose preference signal is dense. Unlike opaque weight updates, the resulting specifications are human-readable and double as interpretable and transparent written embodiments of the preference signal that produced them.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
