Conversation Forests：微调大型语言模型以用于多轮医疗对话的关键是分支
Conversation Forests: The Key to Fine Tuning Large Language Models for Multi-Turn Medical Conversations is Branching
AI总结：
针对DPO和GRPO等方法在多轮医疗对话中表现不足的问题，提出强化学习框架SCF，利用分支对话架构生成多轮延续以微调LLMs，实验证明其诊断准确性优于线性架构。
摘要：
Direct Preference Optimization (DPO) 和 Group Relative Policy Optimization (GRPO) 等微调方法在训练大型语言模型 (LLMs) 执行单轮任务方面已取得成功。然而，这些方法在多轮应用中表现不足，例如诊断性患者访谈，在这些应用中，理解早期对话轮次如何影响下游的补全和结果至关重要。在医学领域，多轮视角对于学习诊断图式和更好地理解对话动态至关重要。为了弥补这一差距，我引入了 Savage Conversation Forests (SCF)，这是一个强化学习框架，利用分支对话架构来微调 LLMs 以进行多轮对话。SCF 在每一轮生成多个可能的对话延续，使模型能够学习不同的早期响应如何影响下游交互和诊断结果。在模拟医患对话的实验中，带有分支的 SCF 在诊断准确性上优于线性对话架构。我假设 SCF 的改进源于其能够在对话轮次之间提供更丰富、相互依赖的训练信号。这些结果表明，分支训练架构是在复杂的多轮对话任务中微调 LLMs 的重要策略。
英文摘要：
Fine-tuning methods such as Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO) have demonstrated success in training large language models (LLMs) for single-turn tasks. However, these methods fall short in multi-turn applications, such as diagnostic patient interviewing, where understanding how early conversational turns influence downstream completions and outcomes is essential. In medicine, a multi-turn perspective is critical for learning diagnostic schemas and better understanding conversation dynamics. To address this gap, I introduce Savage Conversation Forests (SCF), a reinforcement learning framework that leverages a branched conversation architecture to fine-tune LLMs for multi-turn dialogue. SCF generates multiple possible conversation continuations at each turn, enabling the model to learn how different early responses affect downstream interactions and diagnostic outcomes. In experiments simulating doctor-patient conversations, SCF with branching outperforms linear conversation architectures on diagnostic accuracy. I hypothesize that SCF's improvements stem from its ability to provide richer, interdependent training signals across conversation turns. These results suggest that a branched training architecture is an important strategy for fine tuning LLMs in complex multi-turn conversational tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
