SPRI：利用情境化原则对齐大型语言模型
SPRI: Aligning Large Language Models with Context-Situated Principles
AI总结：
提出SPRI框架，通过实时自动生成情境化原则来对齐大型语言模型，在复杂任务、评分标准和真实性提升上表现优异，且无需人工干预。
摘要：
对齐大型语言模型以整合和反映人类价值观，尤其是对于需要复杂人类监督的任务，是非常艰巨的，因为依赖人类专业知识提供特定情境的指导既耗费资源又耗时。先前的工作利用预定义的规则或原则集合来引导模型的行为（Bai et al., 2022; Sun et al., 2023）。然而，这些原则往往是通用的，使得将它们适应于每个单独的输入查询或上下文具有挑战性。在本研究中，我们提出了Situated-PRInciples（SPRI），这是一个需要极少或无需人工努力的框架，旨在为每个输入查询实时自动生成指导原则，并利用它们来对齐每个响应。我们在三个任务上评估了SPRI，并表明：1）SPRI能够在复杂的特定领域任务中推导出原则，其性能与专家制定的原则相当；2）SPRI生成的原则产生了特定实例的评分标准，优于先前的LLM-as-a-judge框架；3）使用SPRI生成合成SFT数据在真实性方面带来了实质性改进。我们在https://github.com/honglizhan/SPRI-public发布了我们的代码和模型生成结果。
英文摘要：
Aligning Large Language Models to integrate and reflect human values, especially for tasks that demand intricate human oversight, is arduous since it is resource-intensive and time-consuming to depend on human expertise for context-specific guidance. Prior work has utilized predefined sets of rules or principles to steer the behavior of models (Bai et al., 2022; Sun et al., 2023). However, these principles tend to be generic, making it challenging to adapt them to each individual input query or context. In this work, we present Situated-PRInciples (SPRI), a framework requiring minimal or no human effort that is designed to automatically generate guiding principles in real-time for each input query and utilize them to align each response. We evaluate SPRI on three tasks, and show that 1) SPRI can derive principles in a complex domain-specific task that leads to on-par performance as expert-crafted ones; 2) SPRI-generated principles lead to instance-specific rubrics that outperform prior LLM-as-a-judge frameworks; 3) using SPRI to generate synthetic SFT data leads to substantial improvement on truthfulness. We release our code and model generations at https://github.com/honglizhan/SPRI-public.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
