arXiv 2509.009742025-12-07cs.CL

RPRO：基于优先级的强化优化用于增强医学问答与诊断推理

RPRO: Ranked Preference Reinforcement Optimization for Enhancing Medical QA and Diagnostic Reasoning

Department of Computer Science and Information Engineering, National Taiwan University of Science and Technology（计算机科学与信息工程系，台湾科技大学）；Department of Systems Engineering, Stevens Institute of Technology（系统工程系，史蒂文斯理工学院）；Department of Management Information Systems, National Chengchi University（管理信息系，National Chengchi University）；AI Research Center, Agaruda System（Agaruda系统人工智能研究中心）；Department of Computer Science, Stevens Institute of Technology（计算机科学系，史蒂文斯理工学院）；Far Eastern Memorial Hospital（东部纪念医院）；Smart Healthcare Interdisciplinary College, National Taipei University of Nursing and Health Sciences（智慧医疗跨领域学院，台北大学护理及健康科学学院）

Chia-Hsuan Hsu, Jun-En Ding, Hsin-Ling Hsu, Chih-Ho Hsu, Li-Hung Yao, Chun-Chieh Liao, Feng Liu, Fang-Ming Hung

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

RPRO通过结合强化学习与优先级驱动的推理细化，提升医疗问答和诊断推理的可靠性与准确性。

摘要：

医学问答需要整合领域知识与逻辑推理的高级推理能力。然而，现有大型语言模型（LLMs）往往生成的推理链缺乏事实准确性和临床可靠性。我们提出了Ranked Preference Reinforcement Optimization（RPRO），一种结合强化学习与基于优先级的推理细化的新型框架，以提升临床推理链（CoT）的性能。RPRO区别于先前方法之处在于采用了任务自适应的推理模板和一个概率评估机制，使模型输出与已确立的临床工作流程对齐，同时自动识别并纠正低质量的推理链。与传统成对偏好方法不同，RPRO引入了基于Bradley-Terry模型的组级排名优化，并结合KL散度正则化以实现稳定的训练。在PubMedQA、MedQA-USMLE以及来自Far Eastern Memorial Hospital（FEMH）的现实世界临床数据集上的实验表明，RPRO在强基线模型上表现出一致的改进。值得注意的是，我们的2B参数模型在性能上超过了许多更大的7B-20B模型，包括医疗专用变体。这些发现表明，结合偏好优化与质量驱动的细化可以提供一种可扩展且具有临床依据的方法，用于构建更可靠的医疗LLMs。

英文摘要：

Medical question answering requires advanced reasoning that integrates domain knowledge with logical inference. However, existing large language models (LLMs) often generate reasoning chains that lack factual accuracy and clinical reliability. We propose Ranked Preference Reinforcement Optimization (RPRO), a novel framework that combines reinforcement learning with preference-driven reasoning refinement to enhance clinical chain-of-thought (CoT) performance. RPRO distinguishes itself from prior approaches by employing task-adaptive reasoning templates and a probabilistic evaluation mechanism that aligns model outputs with established clinical workflows, while automatically identifying and correcting low-quality reasoning chains. Unlike traditional pairwise preference methods, RPRO introduces a groupwise ranking optimization based on the Bradley--Terry model and incorporates KL-divergence regularization for stable training. Experiments on PubMedQA, MedQA-USMLE, and a real-world clinical dataset from Far Eastern Memorial Hospital (FEMH) demonstrate consistent improvements over strong baselines. Remarkably, our 2B-parameter model outperforms much larger 7B--20B models, including medical-specialized variants. These findings demonstrate that combining preference optimization with quality-driven refinement provides a scalable and clinically grounded approach to building more reliable medical LLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。