基于专家网络的对话推荐系统强化策略优化
Reinforced Strategy Optimization for Conversational Recommender Systems via Network-of-Experts
AI总结：
针对现有基于LLM的对话推荐系统缺乏显式交互策略优化的问题，提出带专家网络架构的强化策略优化方法，通过层级拆解策略决策并结合LLM奖励模型的强化学习实现策略探索，显著提升交互性能。
摘要：
对话推荐系统（Conversational Recommender Systems, CRSs）旨在通过与用户的多轮自然语言交互提供个性化推荐。鉴于大语言模型（Large Language Models, LLMs）具备强大的交互与推理能力，利用LLM构建CRS近期已成为颇具前景的研究方向。然而，现有基于LLM的方法往往缺乏对交互策略的显式优化，仅依赖统一提示词与LLM的内部知识决定交互方式，可能导致效果未达最优。\n本文提出一种面向CRS的新型强化策略优化（Reinforced Strategy Optimization, RSO）方法，通过专家网络架构将策略驱动的响应决策生成过程拆解为宏观策略规划与微观策略适配两个层级。在宏观层面，规划器（Planner）专家选择宏观交互策略（如推荐、解释、鼓励）；在微观层面，执行器（Actor）专家基于选定的宏观策略生成详细响应，并由提供用户偏好、事实依据等补充信息的辅助专家提供引导。这种层级分解解耦了CRS响应生成中各子任务的优化过程，让每个层级的学习更易处理。\n为解决高质量多轮训练数据稀缺的问题，我们将策略学习建模为强化学习问题，在基于LLM的奖励模型引导下实现自动策略探索。大量实验表明，与当前最优基线方法相比，RSO显著提升了交互性能，证明了显式层级策略优化对CRS的有效性。
英文摘要：
Conversational Recommender Systems (CRSs) aim to provide personalized recommendations through multi-turn natural language interactions with users. Given the strong interaction and reasoning skills of Large Language Models (LLMs), leveraging LLMs for CRSs has recently emerged as a promising direction. However, existing LLM-based methods often lack explicit optimization of interaction strategies, instead relying on unified prompts and the LLM's internal knowledge to decide how to interact, which can lead to suboptimal outcomes. In this paper, we propose a novel Reinforced Strategy Optimization (RSO) method for CRS, which decomposes the process of generating strategy-driven response decisions into the macro-level strategy planning and micro-level strategy adaptation through a network-of-experts architecture. At the macro level, a Planner expert selects macro-level interaction strategies (e.g., recommend, explain, encourage). At the micro level, an Actor expert generates detailed responses conditioned on the selected macro-level strategy, guided by auxiliary experts that provide complementary information such as user preferences and factual grounding. This hierarchical decomposition disentangles the optimization of different sub-tasks involved in CRS response generation, enabling more tractable learning at each level. To address the scarcity of high-quality multi-turn training data, we formulate strategy learning as a reinforcement learning problem, guided by an LLM-based reward model to achieve automatic strategy exploration. Extensive experiments show that RSO significantly improves interaction performance compared to state-of-the-art baselines, demonstrating the effectiveness of explicit hierarchical strategy optimization for CRS.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
