arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.25136cs.CLcs.AIcs.LG
收藏

摩擦性政策优化用于大语言模型:认知干预、风险敏感控制与反思对齐

Frictive Policy Optimization for LLMs: Epistemic Intervention, Risk-Sensitive Control, and Reflective Alignment

Brandeis University(布拉德雷大学);Colorado State University(科罗拉多州立大学)

James Pustejovsky, Nikhil Krishnaswamy

AI总结:

本文提出摩擦性政策优化框架,通过干预管理认知与规范风险,引入摩擦干预分类和统一方法,提升模型的认知能力与对齐性。

摘要:

我们提出摩擦性政策优化(FPO),一种学习语言模型策略的框架,使模型不仅决定说什么,还决定何时如何干预以管理认知和规范风险。不同于标准对齐方法仅优化表面偏好或任务效用,FPO将澄清、验证、挑战、引导和拒绝视为显式控制动作,目的是塑造信念、承诺和不确定性的演变。我们将对齐形式化为一个风险敏感的认知控制问题,在此问题中,干预决策基于其对下游认知质量的预期影响,而非仅基于即时奖励。我们引入了一个紧凑的摩擦干预分类学,一个结构化的摩擦功能,以操作化多种对齐失败模式,并提出一个统一的FPO方法家族,涵盖奖励塑造、偏好配对、群体相对排名和风险条件信任区域。我们进一步提出一个评估框架,通过澄清行为、校准、矛盾修复、拒绝比例性和信息效率直接衡量认知能力。这些结果为学习在结果和认知行为上均对齐的代理提供了正式和算法基础。

英文摘要:

We propose Frictive Policy Optimization (FPO), a framework for learning language model policies that regulate not only what to say, but when and how to intervene in order to manage epistemic and normative risk. Unlike standard alignment methods that optimize surface-level preference or task utility, FPO treats clarification, verification, challenge, redirection, and refusal as explicit control actions whose purpose is to shape the evolution of belief, commitment, and uncertainty over time. We formalize alignment as a risk-sensitive epistemic control problem in which intervention decisions are selected based on their expected effect on downstream epistemic quality rather than on immediate reward alone. We introduce a compact taxonomy of frictive interventions, a structured friction functional that operationalizes multiple alignment failure modes, and a unified family of FPO methods spanning reward shaping, preference pairing, group-relative ranking, and risk-conditioned trust regions. We further propose an evaluation framework that measures epistemic competence directly through clarification behavior, calibration, contradiction repair, refusal proportionality, and information efficiency. Together, these results provide a formal and algorithmic foundation for learning agents that are aligned not only in outcome, but in epistemic conduct.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑