因果性可以系统地解决基准测试之下的“怪物”问题
Causality can systematically address the monsters under the bench(marks)
AI总结:
本文提出用因果性框架系统解决机器学习基准评估中的偏差、泄漏和不可靠失败模式等问题,并通过通用抽象拓扑(CATs)和案例研究展示其价值。
摘要:
有效且可靠的评估对于推动经验机器学习的发展至关重要。然而,通用模型的可获取性日益提高,以及向更复杂、高层次任务不断推进的趋势,使得系统性评估变得更加困难。基准测试受到各种偏差、伪影或数据泄漏的困扰,而模型可能由于探索不足的失败模式而表现出不可靠的行为。对这些“怪物”进行随意处理和不一致的表述,可能导致重复劳动、对结果缺乏信任以及缺乏依据的推断。在这篇立场论文中,我们主张因果性提供了一个理想的框架来系统地应对这些挑战。通过在方法中明确因果假设,我们可以忠实地对现象进行建模,提出具有解释力的可检验假设,并利用有原则的分析工具。为了使因果模型设计更易于使用,我们在因果图中识别出几种有用的通用抽象拓扑(Common Abstract Topologies, CATs),这些拓扑有助于深入理解大语言模型的推理能力。通过一系列案例研究,我们展示了因果性这种精确而实用的语言如何阐明一种方法的优势与局限,并为系统性进展激发新的方法。
英文摘要:
Effective and reliable evaluation is essential for advancing empirical machine learning. However, the increasing accessibility of generalist models and the progress towards ever more complex, high-level tasks make systematic evaluation more challenging. Benchmarks are plagued by various biases, artifacts, or leakage, while models may behave unreliably due to poorly explored failure modes. Haphazard treatments and inconsistent formulations of such "monsters" can contribute to a duplication of efforts, a lack of trust in results, and unsupported inferences. In this position paper, we argue causality offers an ideal framework to systematically address these challenges. By making causal assumptions in an approach explicit, we can faithfully model phenomena, formulate testable hypotheses with explanatory power, and leverage principled tools for analysis. To make causal model design more accessible, we identify several useful Common Abstract Topologies (CATs) in causal graphs which help gain insight into the reasoning abilities in large language models. Through a series of case studies, we demonstrate how the precise yet pragmatic language of causality clarifies the strengths and limitations of a method and inspires new approaches for systematic progress.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。