先模拟后上线：面向1.4亿规模的客户体验AI智能体的生产前仿真
Screen Before You Serve: Simulation for Production Customer Experience AI Agents at 140M Scale
浏览论文内容
中文总结 AI 辅助
提出基于假设的仿真工作流，在部署前筛选客户体验AI智能体，通过合成客户和模拟工具输出支持多步工作流，在Nubank大规模应用中提升tNPS和SSR，实现安全高效的生产改进。
中文摘要 AI 辅助
客户体验（CX）智能体使用工具和大型语言模型来处理客户请求，并引导与组织产品的对话交互。改进这些智能体，尤其是在受监管行业中，十分困难：它们必须检测意图、遵循复杂的操作策略并可靠地使用工具。手动端到端测试覆盖范围有限，而实时实验则会让客户面临可能削弱信任的失败。我们提出了一种基于假设的仿真工作流，用于在部署前筛选候选的客户体验智能体。合成客户对智能体响应做出反应，模拟的工具输出支持多步智能体工作流，而无需调用生产后端。我们在Nubank的卡片配送智能体及其扩展后继者——卡片管理（Nubank在巴西最高流量的聊天支持智能体）上使用了Snowglobe仿真器。在4个已部署版本中，仿真与生产版本级别的二元评估器得分显示出高度相关性。仿真引导的迭代在实时A/B测试中将交易净推荐值（tNPS）提高了36.69分。我们还在超过16,000次模拟对话中筛选了开放权重配置。在随后的实时A/B测试中，所选模型将自助服务率（SSR）提高了8.82个百分点，达到Nubank观察到的最高水平，而tNPS没有统计学显著变化。仿真使得在不暴露客户的情况下广泛探索模型、推理设置和提示成为可能，从而实现了仅通过实时实验难以实现的生产改进。
英文摘要
Customer experience (CX) agents use tools and large language models to address customer requests and guide conversational interactions with an organization's products. Improving these agents, especially in regulated industries, is difficult: they must detect intent, follow complex operational policies and use tools reliably. Manual end-to-end testing offers limited coverage, while live experiments expose customers to failures that can erode trust. We present a hypothesis-driven simulation workflow for screening candidate CX agents before deployment. Synthetic customers react to agent responses and simulated tool outputs enable multi-step agentic workflows without invoking production backends. We use the Snowglobe simulator on Nubank's Card Delivery agent and its expanded successor, Card Management - Nubank's highest-volume chat-support agent in Brazil. Across 4 deployed versions, simulated and production version-level binary evaluator scores show high correlation. Simulation-guided iteration increased transactional net promoter score (tNPS) by 36.69 points in a live A/B test. We also screened open-weight configurations in over 16,000 simulated conversations. In a subsequent live A/B test, the selected model increased self-service rate (SSR) by 8.82 percentage points to the highest level observed at Nubank, with no statistically significant change in tNPS. Simulation made broad exploration of models, reasoning settings, and prompts feasible without customer exposure, enabling production improvements that would have been impractical to pursue through live experimentation alone.
发表机构
- Nubank（努班克）
- Guardrails AI（护栏人工智能）
机构由 AI 辅助整理，请以论文原文为准。
