arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2610.30690cs.CLcs.AIcs.CY
收藏

从临床判断到可核查规则:由临床医生主导的心理健康聊天机器人安全护栏

From Clinical Judgment to Checkable Rules: Clinician-Led Guardrails for Mental-Health Chatbots

University of California, Irvine(加州大学欧文分校)

Chenhan Lyu, Kit Wislocki, Sarah Elizabeth Stoeckl, Maitreyi Sinha, Amir Rahmani

AI总结:

本研究提出由临床医生主导的心理健康聊天机器人安全护栏,经形式验证可核查分歧,对比LLM评判者减少虚假警报,同时明确其成本与不确定性默认设置的影响。

摘要:

部署在心理健康场景中的大语言模型(LLM)可被评估安全故障,但由此产生的判断往往难以检查或解释。常见的安全改进方法依赖提示工程、LLM作为评判者(LLM-as-a-Judge)以及宪法性AI(Constitutional AI)。在本研究中,我们描述了一种由临床医生主导的替代方案:两名临床医生定义、确定并裁定规则库的四项安全检查,该规则库经形式验证确保内部一致性,并作为护栏应用于LLM输出。此处及标题中的“临床医生主导”指该规则库,而非事实提取器或不确定性默认设置,后者为我们的工作成果。该验证层的每一项裁决均可通过一条命名规则和16项事实重新计算。与在模型、版本、量化、规则文本及解码方面匹配但缺少该层计算的988/911事实的LLM评判者相比,最大分歧类别(满足规则类别)具有结构:Llama-3.3-70B评判者在其主要提示下,对65条已包含危机资源的回复中的55条(84.6%)要求提供危机资源,而同一模型作为事实提取器时,在这55条中的54条报告存在已识别资源。该验证层因结构原因不会提出此类要求。我们同样同等突出地报告该评判者更强的方面:在一项参考评分检查上的虚假危机警报更少。我们还报告了测量成本(规范修订将过度拒绝基准的保留边界提示的阻断类别结果从6.0%提升至16.0%,主要通过改变事实提取)、临床医生保留我们不确定性默认设置的决定以及结果分析。本贡献并非更安全的裁决,而是一种护栏,其与评判者的分歧可定位到命名规则和事实,且测量成本可追溯到命名事实和不确定性默认设置。

英文摘要:

Large language models (LLMs) deployed in mental-health contexts can be evaluated for safety failures, but the resulting judgments are often difficult to inspect or explain. Common approaches to improving safety rely on prompt engineering, LLM-as-a-Judge, and Constitutional AI. In this work, we describe a clinician-led alternative in which two clinicians define, finalize, and adjudicate the four safety checks of a rule base, which is formally verified for internal consistency and applied as a guardrail over LLM outputs. Here and in the title, clinician-led refers to this rule base, not to the fact extractor or the uncertainty default, which are ours. Every verdict of the resulting verification layer is recomputable from a named rule and sixteen facts. Against an LLM judge matched in model, revision, quantization, rule text, and decoding but without the layer's computed 988/911 fact, the largest disagreement class, the satisfied-rule class, is structured: the Llama-3.3-70B judge, on its primary prompt, demands a crisis resource from 55 of the 65 replies (84.6%) that already contain one, while the same model, asked as a fact extractor, reports a recognized resource present on 54 of the 55. The verification layer never raises that demand, by construction. We report with equal prominence where this judge is stronger: fewer false crisis alarms on the one reference-scored check. We also report the measured cost (a specification revision raised block-class outcomes on held-out boundary prompts of an over-refusal benchmark from 6.0% to 16.0%, chiefly by changing fact extraction), the clinicians' decision to keep our uncertainty default, and the analysis of the result. The contribution is not a safer verdict. It is a guardrail whose disagreements with the judge localize to named rules and facts and whose measured costs trace to named facts and the uncertainty default.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑