跨不同特征集的黑盒知识迁移

中文摘要 AI 辅助

预训练的黑盒预测函数编码了从海量数据集和大量计算中提炼出的知识。然而，当可用的输入特征与黑盒期望的特征不同时，直接使用是不可行的。我们提出一种将预测知识从黑盒迁移到新的异质输入空间的方法。我们的方法将目标回归函数分解为可迁移组件（可由黑盒提供信息）和不可迁移组件（捕获新空间特有的信息）。我们提出两步神经网络流程，从连接两个输入空间的大量未标记特征对中估计可迁移组件，从有限的标记数据中估计不可迁移组件。我们推导了预测风险界，当不可迁移组件较小或平滑时，该风险界优于不可迁移替代方法的风险界，且该流程可适配这两种情况。在额外条件下，我们的估计器的最坏情况风险的多项式阶严格小于仅从标记数据估计的极小极大风险。我们将该框架扩展到多个黑盒（每个黑盒对应自身的输入空间），并表明聚合可相对于最佳单个黑盒降低预测误差。模拟数据和真实数据验证了该方法的实用价值。

英文摘要

Pre-trained black-box predictive functions encode knowledge distilled from massive datasets and extensive computation. However, when the available input features differ from those the black box expects, direct use is infeasible. We introduce a method for transferring predictive knowledge from the black box to a new, heterogeneous input space. Our approach decomposes the target regression function into a transferable component, which the black box can inform, and a non-transferable component, which captures information unique to the new space. We propose a two-step neural network procedure, estimating the transferable component from abundant unlabeled feature pairs that bridge the two input spaces and the non-transferable component from limited labels. We derive prediction risk bounds that improve on those of a non-transfer alternative when the non-transferable component is small or smooth, and the procedure adapts to either case. Under additional conditions, the worst-case risk of our estimator is of strictly smaller polynomial order than the minimax risk of estimation from the labeled data alone. We extend the framework to multiple black boxes, each on its own input space, and show that aggregation can reduce prediction error relative to the best single black box. Simulated and real data demonstrate the practical value of the method.

发表机构

• The Ohio State University（俄亥俄州立大学）
• University of Texas at San Antonio（德克萨斯大学圣安东尼奥分校）

机构由 AI 辅助整理，请以论文原文为准。