实验加速器:基于内容感知排序的A/B测试可解释洞察与创意推荐
Experimentation Accelerator: Interpretable Insights and Creative Recommendations for A/B Testing with Content-Aware ranking
AI总结:
本文提出一个统一框架,通过内容感知排序和可解释洞察,提升A/B测试的效率和创意推荐质量。
摘要:
现代在线实验面临两大瓶颈:稀缺的流量迫使必须在哪些变体进行测试上做出艰难选择,而事后洞察提取是手动的、不一致的,且往往缺乏内容感知。同时,组织未充分利用历史A/B结果和丰富的内容嵌入,这些信息本可以指导优先级排序和创意迭代。我们提出一个统一的框架,旨在(i)优先选择哪些变体进行测试,(ii)解释为何胜者获胜,以及(iii)揭示针对新、高潜力变体的有针对性机会。利用处理嵌入和历史结果,我们训练了一个CTR排序模型,使用固定效应进行上下文转换,以评分候选者并平衡价值和内容多样性。为了更好地可解释性和理解,我们将处理投影到经过精心挑选的语义营销属性上,并通过一个符号一致、稀疏约束的Lasso重新表达排序器,从而获得每个属性的系数和带符号贡献,用于可视化解释、前k驱动因素和自然语言洞察。然后,我们计算一个机会指数,结合属性重要性(来自排序器)与当前实验中的低表达程度,以标记缺失的、高影响属性。最后,LLMs将排序的机会转化为具体的创意建议,并估算学习和转化潜力,从而实现更快、更信息丰富和更高效的测试周期。这些组件已集成到一个名为“Experimentation Accelerator”的真实Adobe产品中,为客户提供基于AI的洞察和机会,以扩展实验规模。我们通过Adobe业务客户的一些真实世界实验评估了所提出框架的性能,以验证生成流水线的高质量。
英文摘要:
Modern online experimentation faces two bottlenecks: scarce traffic forces tough choices on which variants to test, and post-hoc insight extraction is manual, inconsistent, and often content-agnostic. Meanwhile, organizations underuse historical A/B results and rich content embeddings that could guide prioritization and creative iteration. We present a unified framework to (i) prioritize which variants to test, (ii) explain why winners win, and (iii) surface targeted opportunities for new, higher-potential variants. Leveraging treatment embeddings and historical outcomes, we train a CTR ranking model with fixed effects for contextual shifts that scores candidates while balancing value and content diversity. For better interpretability and understanding, we project treatments onto curated semantic marketing attributes and re-express the ranker in this space via a sign-consistent, sparse constrained Lasso, yielding per-attribute coefficients and signed contributions for visual explanations, top-k drivers, and natural-language insights. We then compute an opportunity index combining attribute importance (from the ranker) with under-expression in the current experiment to flag missing, high-impact attributes. Finally, LLMs translate ranked opportunities into concrete creative suggestions and estimate both learning and conversion potential, enabling faster, more informative, and more efficient test cycles. These components have been built into a real Adobe product, called \textit{Experimentation Accelerator}, to provide AI-based insights and opportunities to scale experimentation for customers. We provide an evaluation of the performance of the proposed framework on some real-world experiments by Adobe business customers that validate the high quality of the generation pipeline.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。