在多轮强化学习中调和密集和稀疏信号：面向工业销售代理的双 horizon 信用分配
Harmonizing Dense and Sparse Signals in Multi-turn RL: Dual-Horizon Credit Assignment for Industrial Sales Agents
AI总结：
本文提出双 horizon 信用分配框架，通过分离回合级和会话级奖励优化，提升工业销售代理在转化率、语言流畅性和身份检测率方面的性能。
摘要：
优化大型语言模型用于工业销售需要在长期商业目标（例如转化率）与即时语言约束（如流畅性和合规性）之间取得平衡。传统强化学习往往将这些异质目标合并为一个奖励，导致高幅度的会话级奖励淹没更微妙的回合级信号，从而导致训练不稳定或奖励黑客问题。为了解决这一问题，我们提出了双 horizon 信用分配（DuCA），一个框架，它在时间尺度上分离优化。其核心，独立于 horizon 的优势归一化（HIAN），在融合前分别归一化回合级和会话级奖励的优势，确保来自即时和长期目标的梯度贡献平衡到策略更新。与高保真用户模拟器的广泛实验表明，DuCA优于最先进的GRPO基线，实现了6.82%的转化率相对提升，减少了82.28%的句子重复，降低了27.35%的身份检测率，表明对于需要平衡战略性能和自然语言生成双重需求的工业销售场景，取得了显著改进。
英文摘要：
Optimizing large language models for industrial sales requires balancing long-term commercial objectives (e.g., conversion rate) with immediate linguistic constraints such as fluency and compliance. Conventional reinforcement learning often merges these heterogeneous goals into a single reward, causing high-magnitude session-level rewards to overwhelm subtler turn-level signals, which leads to unstable training or reward hacking. To address this issue, we propose Dual-Horizon Credit Assignment (DuCA), a framework that disentangles optimization across time scales. Its core, Horizon-Independent Advantage Normalization (HIAN), separately normalizes advantages from turn-level and session-level rewards before fusion, ensuring balanced gradient contributions from both immediate and long-term objectives to the policy update. Extensive experiments with a high-fidelity user simulator show DuCA outperforms the state-of-the-art GRPO baseline, achieving a 6.82% relative improvement in conversion rate, reducing inter-sentence repetition by 82.28%, and lowering identity detection rate by 27.35%, indicating a substantial improvement for an industrial sales scenario that effectively balances the dual demands of strategic performance and naturalistic language generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
