arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.08645cs.LGstat.ML
收藏

高效条件不变表示学习

Efficient Conditionally Invariant Representation Learning

Gatsby Unit, UCL(伦敦大学学院盖茨比单元);UBC(不列颠哥伦比亚大学);DeepMind(深度思维);Amii(阿尔伯塔机器智能研究所);UChicago(芝加哥大学);Google Brain(谷歌大脑)

Roman Pogodin, Namrata Deka, Yazhe Li, Danica J. Sutherland, Victor Veitch, Arthur Gretton

AI总结:

本研究提出条件独立回归协方差CIRCE作为条件独立性度量,仅需一次预计算岭回归即可实现特征的条件独立约束,在域不变学习等场景的基准测试中性能优于现有方法。

摘要:

我们提出了条件独立回归协方差(Conditional Independence Regression CovariancE,CIRCE),这是一种用于多变量连续取值变量的条件独立性度量。在我们希望学习数据$X$的神经特征$φ(X)$以估计目标$Y$,同时使特征在给定$Y$的条件下与干扰变量$Z$条件独立的场景中,CIRCE可作为正则化器使用。$Z$和$Y$均被假设为连续取值但维度相对较低,而$X$及其特征可能是复杂且高维的。相关应用场景包括域不变学习、公平性和因果学习。该方法仅需执行一次从$Y$到$Z$核化特征的岭回归,这一步可预先完成。随后只需强制$φ(X)$与该回归的残差相互独立即可,这一过程具备良好的估计特性和一致性保证。相比之下,早期的条件特征依赖度量在特征学习的每一步都需要多次回归,会导致更严重的偏差和方差,以及更高的计算成本。当使用足够丰富的特征时,我们证明了CIRCE为零当且仅当$φ(X) \perp \\\\\\\\!\\\\\\\\! \perp Z \mid Y$。在实验中,我们在多个具有挑战性的基准测试上展现出优于现有方法的性能,包括学习条件不变的图像特征。

英文摘要:

We introduce the Conditional Independence Regression CovariancE (CIRCE), a measure of conditional independence for multivariate continuous-valued variables. CIRCE applies as a regularizer in settings where we wish to learn neural features $φ(X)$ of data $X$ to estimate a target $Y$, while being conditionally independent of a distractor $Z$ given $Y$. Both $Z$ and $Y$ are assumed to be continuous-valued but relatively low dimensional, whereas $X$ and its features may be complex and high dimensional. Relevant settings include domain-invariant learning, fairness, and causal learning. The procedure requires just a single ridge regression from $Y$ to kernelized features of $Z$, which can be done in advance. It is then only necessary to enforce independence of $φ(X)$ from residuals of this regression, which is possible with attractive estimation properties and consistency guarantees. By contrast, earlier measures of conditional feature dependence require multiple regressions for each step of feature learning, resulting in more severe bias and variance, and greater computational cost. When sufficiently rich features are used, we establish that CIRCE is zero if and only if $φ(X) \perp \!\!\! \perp Z \mid Y$. In experiments, we show superior performance to previous methods on challenging benchmarks, including learning conditionally invariant image features.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑