arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.11661cs.AI
收藏

夸克医疗对齐:一种全面的多维对齐与协作优化范式

Quark Medical Alignment: A Holistic Multi-Dimensional Alignment and Collaborative Optimization Paradigm

Qwen Applications Business Group, Alibaba(阿里巴巴通义应用事业群);Peking University(北京大学)

Tianxiang Xu, Jiayi Liu, Yixuan Tong, Jialu Xu, Yunqing Wei, Kaiwen Feng, PanPan Hou, Kangping Yin, Jiyuan Hu, Hao Zhou, Zhenxin Ma, Jian Xu, Guanjun Jiang

AI总结:

本文提出夸克医疗对齐范式,通过多维对齐矩阵和统一优化机制解决医疗问答中的对齐问题,实现正确性、安全性和合规性的协同优化。

摘要:

尽管近年来大型语言模型对齐的强化学习取得了快速进展,但将其范式转移到高风险的医疗问答中却揭示了根本性的范式不匹配。强化学习从人类反馈依赖于昂贵且往往无法反映医学事实绝对正确性的偏好注释。强化学习从可验证奖励缺乏有效的自动验证器,并且难以处理复杂的临床情境。与此同时,医疗对齐需要同时优化正确性、安全性和合规性,然而多目标异质奖励信号容易出现尺度不匹配和优化冲突。为了解决这些挑战,我们提出了一个稳健的医疗对齐范式。我们首先构建了一个全面的多维医疗对齐矩阵,将对齐目标分解为四个类别:基础能力、专家知识、在线反馈和格式规范。在每个类别中,我们建立了一个闭环,其中可观察的指标指导可归因的诊断,从而驱动可优化的奖励,从而为后续的迭代优化提供细粒度、高分辨率的监督信号。为了解决由异质信号引起的梯度主导和优化不稳定性问题,我们进一步提出了统一的优化机制。该机制采用参考冻结归一化来对齐奖励尺度,并实施三因素自适应动态加权策略,以实现以弱点为导向、以风险为优先、以冗余为减少的协作优化。实验结果表明,我们提出的范式在真实世界医疗场景评估中表现出有效性,建立了在垂直领域复杂对齐中的新范式。

英文摘要:

While reinforcement learning for large language model alignment has progressed rapidly in recent years, transferring these paradigms to high-stakes medical question answering reveals a fundamental paradigm mismatch. Reinforcement Learning from Human Feedback relies on preference annotations that are prohibitively expensive and often fail to reflect the absolute correctness of medical facts. Reinforcement Learning from Verifiable Rewards lacks effective automatic verifiers and struggles to handle complex clinical contexts. Meanwhile, medical alignment requires the simultaneous optimization of correctness, safety, and compliance, yet multi-objective heterogeneous reward signals are prone to scale mismatch and optimization conflicts. To address these challenges, we propose a robust medical alignment paradigm. We first construct a holistic multi-dimensional medical alignment matrix that decomposes alignment objectives into four categories: fundamental capabilities, expert knowledge, online feedback, and format specifications. Within each category, we establish a closed loop of where observable metrics inform attributable diagnosis, which in turn drives optimizable rewards, thereby providing fine-grained, high-resolution supervision signals for subsequent iterative optimization. To resolve gradient domination and optimization instability problem caused by heterogeneous signals, we further propose a unified optimization mechanism. This mechanism employs Reference-Frozen Normalization to align reward scales and implements a Tri-Factor Adaptive Dynamic Weighting strategy to achieve collaborative optimization that is weakness-oriented, risk-prioritized, and redundancy-reducing. Experimental results demonstrate the effectiveness of our proposed paradigm in real-world medical scenario evaluations, establishing a new paradigm for complex alignment in vertical domains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑