思考型治疗师：利用监督微调与优势比策略优化训练大语言模型以实施接纳与承诺疗法
The Thinking Therapist: Training Large Language Models to Deliver Acceptance and Commitment Therapy using Supervised Fine-Tuning and Odds Ratio Policy Optimization
AI总结：
本研究通过监督微调与优势比策略优化训练Llama-3.2-3b-Instruct以实施接纳与承诺疗法，发现ORPO在治疗忠实度与共情上显著优于SFT，而思维链仅对SFT模型有益，表明显式推理的效用取决于训练范式。
摘要：
接纳与承诺疗法（ACT）是一种第三波认知行为疗法，在多种精神疾病中已显示出疗效证据。本研究考察了后训练方法和显式推理对小型开放权重的大语言模型（LLM）实施ACT能力的影响。利用Mistral-Large生成的合成ACT会话记录，我们采用两种不同方法训练Llama-3.2-3b-Instruct：监督微调（SFT）和优势比策略优化（ORPO），每种方法分别设置包含和不包含显式思维链（COT）推理步骤。通过将这四个后训练变体与基础Instruct模型进行比较来评估性能。这些模型在模拟治疗会话中进行了基准测试，性能由一个在人类评估上微调过的LLM评判员根据ACT忠实度量表（ACT-FM）和治疗师共情量表（TES）进行定量评估。我们的研究结果表明，ORPO训练的模型在ACT忠实度（$χ^2(5) = 185.15, p < .001$）和治疗共情（$χ^2(5) = 140.37, p < .001$）上均显著优于SFT和Instruct对应模型。COT的效果是有条件的，它为SFT模型提供了显著收益，使ACT-FM分数平均提高2.68分（$p < .001$），但对更优的ORPO或instruct微调变体没有明显优势。我们认为ORPO的优越性源于其能够学习治疗的“过程”而非模仿“内容”，这是ACT的一个关键方面，而COT则作为仅通过模仿训练的模型的必要支架。本研究证实，偏好对齐的策略优化能够有效地将ACT能力注入小型LLM，且显式推理的效用高度依赖于底层的训练范式。
英文摘要：
Acceptance and Commitment Therapy (ACT) is a third-wave cognitive behavioral therapy with emerging evidence of efficacy in several psychiatric conditions. This study investigates the impact of post-training methodology and explicit reasoning on the ability of a small open-weight large language model (LLM) to deliver ACT. Using synthetic ACT transcripts generated by Mistral-Large, we trained Llama-3.2-3b-Instruct with two distinct approaches, supervised fine-tuning (SFT) and odds ratio policy optimization (ORPO), each with and without an explicit chain-of-thought (COT) reasoning step. Performance was evaluated by comparing these four post-trained variants against the base Instruct model. These models were benchmarked in simulated therapy sessions, with performance quantitatively assessed on the ACT Fidelity Measure (ACT-FM) and the Therapist Empathy Scale (TES) by an LLM judge that had been fine-tuned on human evaluations. Our findings demonstrate that the ORPO-trained models significantly outperformed both their SFT and Instruct counterparts on ACT fidelity ($χ^2(5) = 185.15, p < .001$) and therapeutic empathy ($χ^2(5) = 140.37, p < .001$). The effect of COT was conditional as it provided a significant benefit to SFT models, improving ACT-FM scores by an average of 2.68 points ($p < .001$), while offering no discernible advantage to the superior ORPO or instruct-tuned variants. We posit that the superiority of ORPO stems from its ability to learn the therapeutic `process' over imitating `content,' a key aspect of ACT, while COT acts as a necessary scaffold for models trained only via imitation. This study establishes that preference-aligned policy optimization can effectively instill ACT competencies in small LLMs, and that the utility of explicit reasoning is highly dependent on the underlying training paradigm.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
