arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2302.02224cs.LGstat.ML
收藏

TAP:用于来自无标注模态的跨模态知识迁移的注意力补丁

TAP: The Attention Patch for Cross-Modal Knowledge Transfer from Unlabeled Modality

Northeastern University(东北大学)

Yinsong Wang, Shahin Shahrampour

AI总结:

针对利用无标注未配对次要模态提升主要模态监督学习性能的问题,本文基于NW核回归提出神经网络附加组件TAP,实现跨模态数据级知识迁移,实验证明其可在多场景下显著提升模型泛化能力。

摘要:

本文研究一种跨模态学习框架,其目标是利用无标注、未配对的次要模态来提升主要模态上监督学习的性能。本文采用概率方法进行缺失信息估计,证明次要模态中包含的额外信息可通过Nadaraya-Watson(NW)核回归估计,且该估计形式可进一步表示为(线性变换下的)核化交叉注意力模块。这一表达式为提出The Attention Patch(TAP,注意力补丁)奠定了基础,TAP是一种简单的神经网络附加组件,经训练后可实现来自无标注模态的数据级知识迁移。本文使用真实世界数据集开展了大量数值模拟,结果表明TAP能够利用看似无法使用的无标注跨模态数据,在不同领域和不同神经网络架构上带来统计上显著的泛化性能提升。

英文摘要:

This paper addresses a cross-modal learning framework, where the objective is to enhance the performance of supervised learning in the primary modality using an unlabeled, unpaired secondary modality. Taking a probabilistic approach for missing information estimation, we show that the extra information contained in the secondary modality can be estimated via Nadaraya-Watson (NW) kernel regression, which can further be expressed as a kernelized cross-attention module (under linear transformation). This expression lays the foundation for introducing The Attention Patch (TAP), a simple neural network add-on that can be trained to allow data-level knowledge transfer from the unlabeled modality. We provide extensive numerical simulations using real-world datasets to show that TAP can provide statistically significant improvement in generalization across different domains and different neural network architectures, making use of seemingly unusable unlabeled cross-modal data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑