基础模型研究的有效性威胁
Validity Threats for Foundation Model Research
AI总结:
本文提出一个因果推断评估框架,将基础模型研究中的不同近似实验策略(代理实验、观察性研究、单次运行设计)映射为四种有效性(统计、内部、外部、构念)的权衡,揭示并分析计算节省带来的隐蔽有效性威胁。
摘要:
受控实验是机器学习研究的基石,但在现代基础模型的规模下,它们变得过于昂贵。相反,研究界越来越依赖于以较低成本近似理想实验的研究策略:代理实验和缩放定律、使用公开模型的观察性研究,以及利用单个训练运行内部变化的单次运行设计。在这项工作中,我们认为在计算预算内近似大规模实验没有免费午餐。具体来说,计算节省是以有效性威胁为代价的——隐藏且有时无法检验的假设,当这些假设被违反时,会使研究主张无效。为了帮助应对这些威胁,我们提出了一个评估框架,将基础模型研究视为因果推断问题。在这个框架内,我们通过从经验社会科学中改编的四种有效性——统计、内部、外部和构念有效性——来评估不同的研究策略。我们发现每种策略都有其特有的有效性特征:代理实验以外部和构念有效性换取统计和内部有效性;观察性研究面临混杂和效应异质性;单次运行设计则因处理单元之间的干扰而紧张。这一分析揭示了文献中未得到充分关注的若干有效性威胁。总体而言,我们的评估框架为研究人员提供了一个实用的工具包,用于审视基础模型研究设计中的有效性威胁。
英文摘要:
Controlled experiments are the backbone of machine learning research, but at the scale of modern foundation models, they have become prohibitively expensive. Instead, the community increasingly relies on research strategies that approximate the ideal experiment at a fraction of the cost: proxy experiments and scaling laws, observational studies with publicly available models, and single-run designs that leverage variation within individual training runs. In this work, we argue that there is no free lunch when approximating large-scale experiments on a compute budget. Specifically, savings in compute come at the cost of validity threats -- hidden and sometimes untestable assumptions that, when violated, can invalidate research claims. To help navigate such threats, we propose an evaluation framework that casts foundation model research as a causal inference problem. Within this framework, we evaluate different research strategies through four types of validity adapted from the empirical social sciences -- statistical, internal, external, and construct validity. We find that each strategy comes with a characteristic validity profile: proxy experiments trade external and construct validity for statistical and internal validity; observational studies face confounding and effect heterogeneity; and single-run designs are strained by interference between treated units. This analysis reveals several validity threats that have received insufficient attention in the literature. Overall, our evaluation framework provides researchers with a practical toolkit for scrutinizing validity threats in foundation model research~designs.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。