The predominant de facto paradigm of testing ML models relies on either using only held-out data to compute aggregate evaluation metrics or by assessing the performance on different subgroups. However, such data-only testing methods operate under the restrictive assumption that the available empirical data is the sole input for testing ML models, disregarding valuable contextual information that could guide model testing. In this paper, we challenge the go-to approach of data-only testing and introduce context-aware testing (CAT) which uses context as an inductive bias to guide the search for meaningful model failures. We instantiate the first CAT system, SMART Testing, which employs large language models to hypothesize relevant and likely failures, which are evaluated on data using a self-falsification mechanism. Through empirical evaluations in diverse settings, we show that SMART automatically identifies more relevant and impactful failures than alternatives, demonstrating the potential of CAT as a testing paradigm.
Enhancing LLMs for Graph Tasks via Graph-aware LoRA Generation
State Key Lab. of AI Safety, Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所人工智能安全国家重点实验室);University of Chinese Academy of Sciences(中国科学院大学);DAMO Academy, Alibaba Group(阿里巴巴集团达摩院)
Mate! Are You Really Aware? An Explainability-Guided Testing Framework for Robustness of Malware Detectors
CSIRO’s Data61(澳大利亚联邦科学与工业研究组织数据61研究所);Cybersecurity CRC(网络安全合作研究中心);Hong Kong University of Science and Technology (GZ)(香港科技大学(广州));Shanghai Jiao Tong University(上海交通大学);Peng Cheng Laboratory(鹏城实验室)
Toward Efficient Testing of Graph Neural Networks via Test Input Prioritization
Computer Science and Technology(计算机科学与技术);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Integrated Circuits(集成电路);Zhejiang University(浙江大学);Software Engineering Institute(软件工程研究所);East China Normal University(华东师范大学)
Physics-aware Machine Learning Revolutionizes Scientific Paradigm for Machine Learning and Process-based Hydrology
Technical University of Munich(慕尼黑工业大学);School of Engineering and Design, Technical University of Munich(慕尼黑工业大学工程与设计学院);School of Geographical Sciences, University of Bristol(布里斯托大学地理科学学院);Ludwig-Maximilians-University(路德维希-马克西米利安-慕尼黑大学);Munich Center for Machine Learning(慕尼黑机器学习中心)
Knowledge-Aware Meta-learning for Low-Resource Text Classification
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Stanford University(斯坦福大学);University of Science and Technology of China(中国科学技术大学);Carnegie Mellon University(卡内基梅隆大学)
Uncertainty-Aware Testing-Time Optimization for 3D Human Pose Estimation
State Key Laboratory of General Artificial Intelligence, Peking University, Shenzhen Graduate School(国家一般人工智能实验室,北京大学深圳研究生院);School of Engineering, École Polytechnique Fédérale de Lausanne(瑞士联邦理工学院(洛桑)工程学院);School of Computer and Communication Sciences, École Polytechnique Fédérale de Lausanne(瑞士联邦理工学院(洛桑)计算机与通信科学学院);School of Computer Science and Engineering, Nanyang Technological University(南洋理工大学计算机科学与工程学院);Department of Computer Science, ETH Zurich(苏黎世联邦理工学院计算机科学系);Department of Information, University of Pisa(意大利比萨大学信息系);Department of Information Engineering and Computer Science, University of Trento(意大利特伦托大学信息工程与计算机科学系)