大型语言模型的风险条件微调
Risk-Conditioned Fine-Tuning of Large Language Models
浏览论文内容
中文总结 AI 辅助
本文提出风险条件RLHF框架，训练单一策略以提供连续风险控制接口，使LLM在推理时可灵活调整风险规避程度，无需重训或部署多模型，实验验证其有效性。
中文摘要 AI 辅助
大型语言模型（LLMs）越来越多地被部署在罕见但严重的有害生成可能产生重大后果的环境中。现有的风险规避RLHF通过优化条件风险价值（CVaR）来解决这个问题，但它针对固定的风险水平训练策略，因此无法在推理时调整所需的风险规避程度。在本文中，我们提出了风险条件RLHF，这是一个训练单一策略的框架，提供连续的风险控制接口，使用户无需重新训练或部署多个特定风险的模型，即可选择不同程度的风险规避。跨多个基准的实验表明，单一的风险条件策略可以在推理时适应不同的风险水平，从而实现更灵活和风险感知的LLM部署。
英文摘要
Large Language Models (LLMs) are increasingly deployed in settings where rare but severe harmful generations can have significant consequences. Existing Risk-Averse RLHF addresses this issue by optimizing Conditional Value-at-Risk (CVaR), but it trains policies for fixed risk levels and therefore cannot adjust the desired degree of risk aversion at inference time. In this paper, we propose risk-conditioned RLHF, a framework that trains a single policy that provides a continuous risk-control interface, enabling users to select different degrees of risk aversion without retraining or deploying multiple risk-specific models. Experiments across multiple benchmarks demonstrate that a single risk-conditioned policy can adapt to different risk levels at inference time, enabling more flexible and risk-aware LLM deployment.
发表机构
- Tulane University（杜兰大学）
机构由 AI 辅助整理，请以论文原文为准。
