arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.01036stat.MEcs.LGstat.APstat.ML
收藏

基于提升值排序:大规模A/B测试的成本效益方法

Ranking by Lifts: A Cost-Benefit Approach to Large-Scale A/B Tests

Indian School of Business(印度学校商业学院);The Wharton School University of Pennsylvania(沃顿商学院大学)

Pallavi Basu, Ron Berman

AI总结:

本文针对商业A/B测试中平衡提升值与错误发现成本的问题,提出基于预期提升值与成本比率排序的经验贝叶斯框架,结合贪心背包算法和局部错误发现率,在约束成本加权FDR下最大化利润,且性能优于现有方法。

摘要:

A/B测试是商业实验决策的核心工具,尤其在数字平台和市场领域。从业者常优先考虑性能指标的提升值,同时寻求控制错误发现的成本。本文提出一个决策理论框架,旨在最大化预期利润,同时约束成本加权的错误发现率(FDR)。我们提出一种经验贝叶斯方法,使用贪心背包算法基于预期提升值与成本的比率对实验排序,将局部错误发现率(lfdr)作为关键统计量。由此得到的先知规则有效且排序最优。在大规模场景中,我们证明了数据驱动实现的渐近有效性,并展示其有限样本性能优于现有FDR控制方法。在Optimizely平台上的A/B测试应用突显了该方法的商业价值。

英文摘要:

A/B testing is a core tool for decision-making in business experimentation, particularly in digital platforms and marketplaces. Practitioners often prioritize lift in performance metrics while seeking to control the costs of false discoveries. This paper develops a decision-theoretic framework for maximizing expected profit subject to a constraint on the cost-weighted false discovery rate (FDR). We propose an empirical Bayes approach that uses a greedy knapsack algorithm to rank experiments based on the ratio of expected lift to cost, incorporating the local false discovery rate (lfdr) as a key statistic. The resulting oracle rule is valid and rank-optimal. In large-scale settings, we establish the asymptotic validity of a data-driven implementation and demonstrate superior finite-sample performance over existing FDR-controlling methods. An application to A/B tests run on the Optimizely platform highlights the business value of the approach.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑