RACE-Align：面向大语言模型的检索增强与思维链增强偏好对齐
RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Models
AI总结：
本文提出RACE-Align框架，通过融合检索增强与思维链推理构建偏好数据并用DPO对齐大语言模型，在中医领域显著提升了准确性、推理深度和可解释性。
摘要：
大语言模型（LLMs）在垂直领域中面临准确性、领域特定推理和可解释性方面的挑战。传统的偏好对齐方法，如基于人类反馈的强化学习（RLHF）和直接偏好优化（DPO），往往忽略了底层的知识来源和推理逻辑。本文提出了RACE-Align（Retrieval-Augmented and Chain-of-Thought Enhanced Alignment，检索增强与思维链增强对齐），这是一个旨在解决上述局限性的新型框架。RACE-Align系统性地构建了一个包含外部知识支持和显式思维链（CoT）推理的二元偏好数据集，然后使用DPO算法对大语言模型进行对齐。其核心创新在于偏好数据的构建策略：该策略整合了AI驱动的检索以实现事实依据，从而增强知识性和准确性，并强调对领域特定CoT的优化，将推理过程本身视为一个关键的偏好维度。一个多阶段、AI驱动的精炼流程以低成本高效的方式生成这些偏好对。在中医（TCM）领域使用Qwen3-1.7B作为基座模型进行的实验验证表明，RACE-Align显著优于原始基座模型和仅使用监督微调（SFT）进行微调的模型。在多个维度上均观察到改进，包括答案准确性、信息丰富度、中医思维模式的应用、推理的逻辑性与深度以及可解释性。这些发现表明，RACE-Align为在复杂垂直领域中增强大语言模型的知识应用、推理可靠性和过程透明度提供了一条有效途径。
英文摘要：
Large Language Models (LLMs) struggle with accuracy, domain-specific reasoning, and interpretability in vertical domains. Traditional preference alignment methods like Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) often overlook the underlying knowledge sources and reasoning logic. This paper introduces RACE-Align (Retrieval-Augmented and Chain-of-Thought Enhanced Alignment), a novel framework designed to address these limitations. RACE-Align systematically constructs a binary preference dataset incorporating external knowledge support and explicit Chain-of-Thought (CoT) reasoning, then aligns LLMs using the DPO algorithm. The core innovation lies in its preference data construction strategy: it integrates AI-driven retrieval for factual grounding, enhancing knowledgeability and accuracy, and emphasizes the optimization of domain-specific CoT, treating the reasoning process itself as a key preference dimension. A multi-stage, AI-driven refinement pipeline cost-effectively generates these preference pairs. Experimental validation in Traditional Chinese Medicine (TCM) using Qwen3-1.7B as the base model demonstrates that RACE-Align significantly outperforms the original base model and a model fine-tuned only with Supervised Fine-Tuning (SFT). Improvements were observed across multiple dimensions, including answer accuracy, information richness, application of TCM thinking patterns, logicality and depth of reasoning, and interpretability. These findings suggest RACE-Align offers an effective pathway to enhance LLMs' knowledge application, reasoning reliability, and process transparency in complex vertical domains.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
