LiveThinking:通过强化学习赋能AI驱动的直播实时高效推理
LiveThinking: Enabling Real-Time Efficient Reasoning for AI-Powered Livestreaming via Reinforcement Learning
AI总结:
针对AI电商直播的高延迟问题,提出两阶段框架LiveThinking,通过模型蒸馏和强化学习压缩推理路径,实现亚秒级延迟并显著提升互动表现与GMV。
摘要:
在AI驱动的电商直播中,数字人需要实时响应以提升互动,而高延迟的大型推理模型(LRM)并不适合此任务。我们引入了LiveThinking,一个弥合此差距的实用两阶段优化框架。首先,我们通过拒绝采样微调(RFT)将670B的教师LRM蒸馏为轻量级的30B混合专家(MoE)模型(3B活跃),以解决计算成本问题。这降低了部署开销,但保留了教师模型冗长的推理过程,导致延迟。为解决此问题,我们的第二阶段采用强化学习与组相对策略优化(GRPO)来压缩模型的推理路径,并由平衡正确性、有用性和简洁性的多目标奖励函数引导。LiveThinking实现了30倍的计算成本降低,实现亚秒级延迟。在淘宝直播的实际应用中,它将响应正确性提高了3.3%,有用性提高了21.8%。经数十万观众测试,我们的系统导致商品交易总额(GMV)出现统计学显著增长,证明了其在提升实时互动场景中的用户体验和商业表现的有效性。
英文摘要:
In AI-powered e-commerce livestreaming, digital avatars require real-time responses to drive engagement, a task for which high-latency Large Reasoning Models (LRMs) are ill-suited. We introduce LiveThinking, a practical two-stage optimization framework to bridge this gap. First, we address computational cost by distilling a 670B teacher LRM into a lightweight 30B Mixture-of-Experts (MoE) model (3B active) using Rejection Sampling Fine-Tuning (RFT). This reduces deployment overhead but preserves the teacher's verbose reasoning, causing latency. To solve this, our second stage employs reinforcement learning with Group Relative Policy Optimization (GRPO) to compress the model's reasoning path, guided by a multi-objective reward function balancing correctness, helpfulness, and brevity. LiveThinking achieves a 30-fold reduction in computational cost, enabling sub-second latency. In real-world application on Taobao Live, it improved response correctness by 3.3% and helpfulness by 21.8%. Tested by hundreds of thousands of viewers, our system led to a statistically significant increase in Gross Merchandise Volume (GMV), demonstrating its effectiveness in enhancing user experience and commercial performance in live, interactive settings.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。