基于提升值排序:大规模A/B测试的成本效益方法
Ranking by Lifts: A Cost-Benefit Approach to Large-Scale A/B Tests
AI总结:
本文针对商业A/B测试中平衡提升值与错误发现成本的问题,提出基于预期提升值与成本比率排序的经验贝叶斯框架,结合贪心背包算法和局部错误发现率,在约束成本加权FDR下最大化利润,且性能优于现有方法。
摘要:
A/B测试是商业实验决策的核心工具,尤其在数字平台和市场领域。从业者常优先考虑性能指标的提升值,同时寻求控制错误发现的成本。本文提出一个决策理论框架,旨在最大化预期利润,同时约束成本加权的错误发现率(FDR)。我们提出一种经验贝叶斯方法,使用贪心背包算法基于预期提升值与成本的比率对实验排序,将局部错误发现率(lfdr)作为关键统计量。由此得到的先知规则有效且排序最优。在大规模场景中,我们证明了数据驱动实现的渐近有效性,并展示其有限样本性能优于现有FDR控制方法。在Optimizely平台上的A/B测试应用突显了该方法的商业价值。
英文摘要:
A/B testing is a core tool for decision-making in business experimentation, particularly in digital platforms and marketplaces. Practitioners often prioritize lift in performance metrics while seeking to control the costs of false discoveries. This paper develops a decision-theoretic framework for maximizing expected profit subject to a constraint on the cost-weighted false discovery rate (FDR). We propose an empirical Bayes approach that uses a greedy knapsack algorithm to rank experiments based on the ratio of expected lift to cost, incorporating the local false discovery rate (lfdr) as a key statistic. The resulting oracle rule is valid and rank-optimal. In large-scale settings, we establish the asymptotic validity of a data-driven implementation and demonstrate superior finite-sample performance over existing FDR-controlling methods. An application to A/B tests run on the Optimizely platform highlights the business value of the approach.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。