TaoSR-AGRL：用于电子商务搜索相关性的自适应引导强化学习框架
TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance
AI总结：
电商搜索中查询-产品相关性预测很关键，现有方法有局限。提出TaoSR-AGRL框架，通过规则感知奖励塑造和自适应引导重放两大创新，提升模型推理能力，在实验中表现优于基线，已成功部署。
摘要：
查询-产品相关性预测对电子商务搜索至关重要，在人工智能购物时代变得更加关键。大语言模型启用了基于生成和推理的方法，通常通过监督微调（SFT）或直接偏好优化（DPO）等偏好优化方法进行对齐。然而，业务规则和用户查询的复杂性不断增加，暴露了现有方法无法赋予模型强大的长尾和具有挑战性的案例的推理能力。通过诸如组相对策略优化（GRPO）等强化学习策略来解决这一问题的努力往往受到稀疏终端奖励的困扰，为多步推理提供的指导不足，并且收敛速度较慢。为了应对这些挑战，我们提出了TaoSR-AGRL，这是一种用于淘宝搜索相关性中基于大语言模型的相关性预测的自适应引导强化学习框架。TaoSR-AGRL引入了两个关键创新：（1）规则感知奖励塑造，它将最终相关性判断分解为与特定领域相关性标准对齐的密集、结构化奖励；（2）自适应引导重放，它在训练期间识别低准确性的展开，并注入有针对性的真实指导，以引导策略远离停滞、违反规则的推理模式，转向合规轨迹。TaoSR-AGRL在大规模真实世界数据集上进行了评估，并通过淘宝搜索上的在线并行人工评估进行了评估。它在离线实验中始终优于DPO和标准GRPO基线，提高了相关性准确性、规则遵守和训练稳定性。使用TaoSR-AGRL训练的模型已成功部署在淘宝的主要搜索场景中，为数亿用户提供服务。
英文摘要：
Query-product relevance prediction is fundamental to e-commerce search and has become even more critical in the era of AI-powered shopping, where semantic understanding and complex reasoning directly shape the user experience and business conversion. Large Language Models (LLMs) enable generative, reasoning-based approaches, typically aligned via supervised fine-tuning (SFT) or preference optimization methods like Direct Preference Optimization (DPO). However, the increasing complexity of business rules and user queries exposes the inability of existing methods to endow models with robust reasoning capacity for long-tail and challenging cases. Efforts to address this via reinforcement learning strategies like Group Relative Policy Optimization (GRPO) often suffer from sparse terminal rewards, offering insufficient guidance for multi-step reasoning and slowing convergence. To address these challenges, we propose TaoSR-AGRL, an Adaptive Guided Reinforcement Learning framework for LLM-based relevance prediction in Taobao Search Relevance. TaoSR-AGRL introduces two key innovations: (1) Rule-aware Reward Shaping, which decomposes the final relevance judgment into dense, structured rewards aligned with domain-specific relevance criteria; and (2) Adaptive Guided Replay, which identifies low-accuracy rollouts during training and injects targeted ground-truth guidance to steer the policy away from stagnant, rule-violating reasoning patterns toward compliant trajectories. TaoSR-AGRL was evaluated on large-scale real-world datasets and through online side-by-side human evaluations on Taobao Search. It consistently outperforms DPO and standard GRPO baselines in offline experiments, improving relevance accuracy, rule adherence, and training stability. The model trained with TaoSR-AGRL has been successfully deployed in the main search scenario on Taobao, serving hundreds of millions of users.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
