基于代理的大规模A/B测试预估测量
Calibrate Globally, Measure Everywhere: Scaling LLM-Based Prevalence Measurement Across A/B Experiments
AI总结：
本文提出了一种基于代理的普及率测量框架，通过离线校准和日志数据实现大规模A/B测试中的高效普及率估计。
摘要：
在线媒体平台经常需要测量用户接触特定内容属性的频率，以评估A/B实验中的权衡。直接方法是采样内容，使用高质量的评分标准（例如专家审核的LLM提示）进行标注，并估计印象加权的普及率。然而，反复运行此类标注以每个实验臂和细分进行，成本太高且速度太慢，无法作为大规模的默认测量方法。我们提出了一种可扩展的基于代理的普及率测量框架，该框架将昂贵的标注与每次实验的评估解耦。该框架在离线期间校准代理信号以参考标签，并随后仅使用印象日志来估计任意实验臂和细分的普及率。我们通过评分分桶作为代理来实现该框架：我们将模型评分离散化为桶，从离线标注样本中估计桶级普及率，并将这些校准的桶级普及率与每个臂的桶分布相结合，以获得快速的日志基于估计。在多个大规模A/B测试中，我们验证了代理估计与参考估计在臂级普及率和治疗-对照差异方面非常接近。这使得在实验中无需每次实验都进行标注任务即可实现可扩展的低延迟普及率测量。
英文摘要：
Online media platforms track the share of impressions associated with content attributes, or prevalence, to evaluate trade-offs and set guardrails in A/B experiments. LLM-based labeling provides a high-fidelity reference measurement, but is cost-prohibitive to run per experiment, per arm, per segment, and per day on a platform with hundreds of concurrent experiments. We describe a surrogate-based prevalence measurement system deployed in Pinterest's experimentation platform. The contribution is system-level rather than estimator-level: the system maintains a single global calibration of ML score buckets, continuously refreshed from a recurring LLM-labeled stream, and reuses the resulting bucket-level prevalences across every experiment via a per-experiment SQL metric and a delta-focused dashboard. Because the calibration is derived from the platform's daily-batch prevalence samples, it remains representative of production traffic as distributions drift, and in Pinterest's deployment it incurs zero incremental labeling cost. Teams without such infrastructure can instantiate the same pattern with a recurring calibration-labeling workflow whose cost is amortized across all downstream experiments rather than paid per experiment, arm, segment, and day. The system serves ~100 experiments and ~250 arms per day across six calibrated content categories, including a holdout program. Relative to per-experiment LLM labeling, which in practice yields a one-shot read per arm on a small subset of experiments, the surrogate provides daily per-arm prevalence on over 20$\times$ as many concurrent arms under the same labeling budget. Across roughly 300 production audits, the surrogate's 95\% confidence interval contains the LLM-based reference point estimate in 92\% of evaluations, and day-level delta aggregation recovers 2--5\% relative shifts that no single per-arm LLM measurement can detect.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
