arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.04531cs.LG
收藏

AExGym:面向自适应实验的基准与环境

AExGym: Benchmarks and Environments for Adaptive Experimentation

Columbia University(哥伦比亚大学);Columbia Business School(哥伦比亚商学院);Meta;University of Pittsburgh(匹兹堡大学)

Jimmy Wang, Ethan Che, Daniel R. Jiang, Hongseok Namkoong

AI总结:

本文提出基于真实数据的自适应实验基准与开源库 AExGym,聚焦非平稳性、延迟反馈等实际挑战,以推动以鲁棒性为核心的自适应实验方法发展。

摘要:

科学和工业领域的创新通常通过随机试验(即 A/B 测试)进行评估。虽然这类静态设计简单且稳健,但在检验大量假设时效率低下或不可行。自适应设计在理论上可以大幅提升统计功效,但由于实践中的脆弱性,其采用范围有限。我们基于真实世界数据集提出了一个自适应实验基准,突出将自适应性付诸实践所面临的主要挑战:非平稳性、批量/延迟反馈、多重结果与目标以及外部有效性。该基准旨在推动方法学发展,将实际性能(例如稳健性)作为核心关注点,而非人为构造实例上的数学保证。我们发布了一个开源库 AExGym,其设计以模块化和可扩展性为理念,使实验从业者能够开发自定义环境和算法。

英文摘要:

Innovations across science and industry are evaluated using randomized trials (a.k.a. A/B tests). While simple and robust, such static designs are inefficient or infeasible for testing many hypotheses. Adaptive designs can greatly improve statistical power in theory, but they have seen limited adoption due to their fragility in practice. We present a benchmark for adaptive experimentation based on real-world datasets, highlighting prominent practical challenges to operationalizing adaptivity: non-stationarity, batched/delayed feedback, multiple outcomes and objectives, and external validity. Our benchmark aims to spur methodological development that puts practical performance (e.g., robustness) as a central concern, rather than mathematical guarantees on contrived instances. We release an open source library, AExGym, which is designed with modularity and extensibility in mind to allow experimentation practitioners to develop custom environments and algorithms.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑