探索用于医疗对话偏好对齐的基于大语言模型的数据标注策略
Exploring LLM-based Data Annotation Strategies for Medical Dialogue Preference Alignment
AI总结：
本研究针对医疗对话偏好对齐标注依赖专家的问题，提出基于标准化患者考核的评估框架，结合流程图与Constitutional AI设计智能体标注方法，其效果优于现有RLAIF方法及开源医疗对话LLM。
摘要：
本研究探讨了利用AI反馈强化学习（Reinforcement Learning from AI Feedback, RLAIF）技术改进医疗对话模型，旨在解决偏好对齐数据标注的挑战，同时降低对医学专家的依赖。我们认为，当前医疗领域RLAIF研究的主要挑战在于自动化评估方法的局限性，以及难以准确表征医生偏好。为应对这些挑战，我们提出了一种基于标准化患者考核的新型评估框架。该框架旨在客观评估大语言模型（large language models, LLMs）在引导用户和遵循指令方面的有效性，可实现不同模型间的综合比较。此外，我们对使用Constitutional AI算法表达医生偏好的有效方式进行了研究，发现流程图的效果尤为突出。基于这一发现，我们提出了一种创新的基于智能体的偏好数据标注方法。该方法可自主生成适配患者病情的医疗对话流程，展现出强大的泛化能力，且减少了对专家参与的需求。我们的结果表明，在标准化患者考核中，基于智能体的方法优于现有RLAIF标注方法，且在多种测试场景中超越了当前开源医疗对话LLM。
英文摘要：
This research examines the use of Reinforcement Learning from AI Feedback (RLAIF) techniques to improve healthcare dialogue models, with the aim of tackling the challenges of preference-aligned data annotation while reducing the reliance on medical experts. We argue that the primary challenges in current RLAIF research for healthcare are the limitations of automated evaluation methods and the difficulties in accurately representing physician preferences. To address these challenges, we present a new evaluation framework based on standardized patient examinations. This framework is designed to objectively assess the effectiveness of large language models (LLMs) in guiding users and following instructions, enabling a comprehensive comparison across different models. Furthermore, our investigation of effective ways to express physician preferences using Constitutional AI algorithms highlighted the particular effectiveness of flowcharts. Utilizing this finding, we introduce an innovative agent-based approach for annotating preference data. This approach autonomously creates medical dialogue flows tailored to the patient's condition, demonstrates strong generalization abilities, and reduces the need for expert involvement. Our results show that the agent-based approach outperforms existing RLAIF annotation methods in standardized patient examinations and surpasses current open source medical dialogue LLMs in various test scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
