arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.24005cs.LG
收藏

Context-Aware Testing: 一种基于大语言模型的模型测试新范式

Context-Aware Testing: A New Paradigm for Model Testing with Large Language Models

University of Cambridge(剑桥大学)

Paulius Rauba, Nabeel Seedat, Max Ruiz Luyten, Mihaela van der Schaar

AI总结:

针对现有机器学习测试仅依赖数据、忽略上下文信息的局限,提出上下文感知测试(CAT)新范式,构建首个系统SMART Testing,利用大语言模型结合自证伪机制查找模型失效,实验证明其效果优于现有方法。

摘要:

当前主流的机器学习模型测试事实范式要么仅使用留出数据计算聚合评估指标,要么评估模型在不同子群体上的表现。然而,这类仅依赖数据的测试方法基于一个限制性假设:可用的经验数据是测试机器学习模型的唯一输入,忽略了可指导模型测试的宝贵上下文信息。本文对仅用数据测试的常用方法提出质疑,引入上下文感知测试(context-aware testing, CAT),将上下文作为归纳偏置来引导搜寻有意义的模型失效情况。我们实现了首个CAT系统SMART Testing,该系统利用大语言模型假设相关且可能出现的失效情况,并通过自证伪机制在数据上评估这些假设。通过在多种场景下的实证评估,我们表明SMART比其他替代方法能自动识别更多相关且影响更大的失效情况,证明了CAT作为一种测试范式的潜力。

英文摘要:

The predominant de facto paradigm of testing ML models relies on either using only held-out data to compute aggregate evaluation metrics or by assessing the performance on different subgroups. However, such data-only testing methods operate under the restrictive assumption that the available empirical data is the sole input for testing ML models, disregarding valuable contextual information that could guide model testing. In this paper, we challenge the go-to approach of data-only testing and introduce context-aware testing (CAT) which uses context as an inductive bias to guide the search for meaningful model failures. We instantiate the first CAT system, SMART Testing, which employs large language models to hypothesize relevant and likely failures, which are evaluated on data using a self-falsification mechanism. Through empirical evaluations in diverse settings, we show that SMART automatically identifies more relevant and impactful failures than alternatives, demonstrating the potential of CAT as a testing paradigm.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑