Modern AI agents such as large language models are trained on diverse tasks -- translation, code generation, mathematical reasoning, and text prediction -- simultaneously. A key question is how to quantify the influence of each individual training task on performance on a target task, a problem we refer to as task attribution. The direct approach, leave-one-out retraining, measures the effect of removing each task, but is computationally infeasible at scale. An alternative approach that builds surrogate models to predict the performance on a target task for any subset of training tasks has emerged in the recent literature. Prior work focuses on linear surrogate models, which capture first-order relationships but miss nonlinear interactions such as XOR-type effects. In this paper, we first consider a unified task-weighting framework for analyzing task-attribution methods and establish a new connection between linear surrogate models and influence functions via a second-order analysis. Then, we introduce kernel surrogate models, which more effectively represent second-order task interactions. To efficiently learn the kernel surrogate, we develop a gradient-based estimation procedure that leverages a first-order approximation of pretrained models; empirically, this yields accurate surrogate estimates with less than $2\%$ relative error without repeated retraining. Experiments across multiple settings -- including mathematical reasoning in transformers, in-context learning, and multi-objective reinforcement learning -- demonstrate the effectiveness of kernel surrogate models. They achieve a $25\%$ higher correlation with the leave-one-out ground truth than linear surrogates and influence-function baselines, enabling more accurate and scalable task attribution. When used for downstream data selection, kernel surrogate models further yield a $40\%$ improvement in the aforementioned settings.
Towards Surrogate Based Dequantization of Quantum Reinforcement Learning
University of the Basque Country UPV/EHU(巴斯克大学 UPV/EHU);TECNALIA, Basque Research and Technology Alliance (BRTA)(TECNALIA,巴斯克研究与技术联盟(BRTA));Freie Universität Berlin(柏林自由大学);Helmholtz-Zentrum Berlin für Materialien und Energie(亥姆霍兹柏林材料与能源中心);IKERBASQUE, Basque Foundation for Science(IKERBASQUE,巴斯克科学基金会);Basque Center for Applied Mathematics (BCAM)(巴斯克应用数学中心(BCAM));African Institute for Mathematical Sciences (AIMS)(非洲数学科学研究所(AIMS));Stellenbosch University(斯泰伦博斯大学);National Institute for Theoretical and Computational Sciences (NITheCS)(国家理论与计算科学研究所(NITheCS))
Faithful and Efficient Explanations for Neural Networks via Neural Tangent Kernel Surrogate Models
Pacific Northwest National Laboratory(太平洋西北国家实验室);University of California, San Diego(加利福尼亚大学圣迭戈分校);Courant Institute, NYU(纽约大学柯朗研究所);Rutgers University(罗格斯大学);University of Washington(华盛顿大学);University of Texas, El Paso(德克萨斯大学埃尔帕索分校)
Scalable Linearized Laplace Approximation via Surrogate Neural Kernel
Machine Learning Group - Computer Science Department Escuela Politécnica Superior(计算机科学系机器学习小组);Computer Science Department Escuela Politécnica Superior(计算机科学系);Universidad Autónoma de Madrid(马德里自治大学);Universidad Pontificia de Comillas(天主教comillas大学);Aalborg University Copenhagen(哥本哈根奥胡斯大学);Centro de Investigación Avanzada en Física Fundamental Universidad Autónoma de Madrid(马德里自治大学先进物理基础研究中心)
A generalized neural tangent kernel for surrogate gradient learning
Department of Physiology, University of Bern(伯恩大学生理学系);Institute for Applied Mathematics, University of Bonn(波恩大学应用数学研究所);Institute of Genetics, University of Bonn(波恩大学遗传学研究所);Bernstein Center Freiburg, University of Freiburg(弗赖堡大学伯恩斯坦中心)
Minimizing False-Positive Attributions in Explanations of Non-Linear Models
Technical University of Denmark(丹麦技术大学);Technische Universität Berlin(柏林技术大学);Physikalisch-Technische Bundesanstalt, Berlin(柏林物理技术部);Charité – Universitätsmedizin Berlin(柏林查理医院大学医学中心)
Amortized Linear-time Exact Shapley Value for Product-Kernel Methods
Rational Intelligence Lab, CISPA Helmholtz Center for Information Security(理性智能实验室,信息安全中心(CISPA));Department of Computer Science, Vrije Universiteit Amsterdam(阿姆斯特丹自由大学计算机科学系);Epistemic Intelligence & Computation Lab, College of Computing & Data Science, Nanyang Technological University(认知智能与计算实验室,南洋理工大学计算机与数据科学学院)
How Quantum Is the Advantage? A Fair, Calibration- and Noise-Aware Benchmark and Attribution Audit of Quantum Machine Learning for Network Intrusion Detection
Efficient kernel surrogates for neural network-based regression
Pacific Northwest National Laboratory(太平洋西北国家实验室);Stony Brook University(石溪大学);The University of Washington(华盛顿大学);Brown University(布朗大学);The University of Texas, El Paso(德克萨斯大学埃尔帕索分校)
本文研究以共轭核 CK 高效近似神经正切核 NTK,建立相对测试损失界并验证其性能,提出低成本提升 DNN、GPT-2 及物理信息算子网络训练的方案。
Surrogate Modeling of 3D Rayleigh-Benard Convection with Equivariant Autoencoders
Lamarr Institute for Machine Learning and Artifical Intelligence(拉玛尔机器学习与人工智能研究所);Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院);Department of Computer Science at Paderborn University(帕德博恩大学计算机科学系);Department of Mechanical and Aerospace Engineering Center for Statistics and Machine Learning(机械与航空航天工程系统计与机器学习中心);Princeton University(普林斯顿大学);Department of Computer Science at TU Dortmund University(杜伊斯堡-艾瑟恩理工大学计算机科学系)