基于多随机掩码自编码器的概率超图用于半监督多模态多任务学习
Probabilistic Hyper-Graphs using Multiple Randomly Masked Autoencoders for Semi-supervised Multi-modal Multi-task Learning
AI总结：
本文提出PHG-MAE模型，结合神经图和掩码自编码器，通过随机掩码多模态数据提升多任务学习性能，并公开了相关工具和数据集。
摘要：
计算机视觉领域在许多模态中拥有大量数据方面受益匪浅，从而在各种视觉任务上取得进步。最近，通过掩码自编码器（MAE）进行自监督预训练的方法受到了广泛关注，通常用作在下游任务（如分类或回归）优化之前的初步步骤。这非常有用，因为它不需要任何手动标注的数据。在本工作中，我们引入了基于掩码自编码器的概率超图（PHG-MAE）：一种新的模型，它在共同的理论框架下将传统的神经图工作与现代的掩码自编码器方法结合起来。通过随机掩码整个模态，而不是仅仅掩码片段，模型在每次前向传递中从每个超边的分布中采样。此外，该模型通过将预训练和微调结合到一个训练循环中，调整了标准的MAE算法。此外，我们的方法使能够创建推理时的集成，通过聚合，提高最终预测性能和一致性。最后，我们展示了可以在集成之上应用知识蒸馏，即使在参数少于100万的模型上，也能有很小的性能损失。尽管我们的工作主要集中在户外无人机场景，这些场景包含多种世界解释和模态，但相同的步骤也可以在其他类似领域中进行，如自动驾驶或室内机器人。为了简化将外部预训练专家整合到计算机视觉多模态多任务学习（MTL）场景中的过程，我们开发了一种数据管道软件。使用这个工具，我们已经创建并发布了Dronescapes数据集的完全自动化扩展。所有技术细节、代码和复现步骤均已公开发布。
英文摘要：
The computer vision domain has greatly benefited from an abundance of data across many modalities to improve on various visual tasks. Recently, there has been a lot of focus on self-supervised pre-training methods through Masked Autoencoders (MAE) \cite{he2022masked,bachmann2022multimae}, usually used as a first step before optimizing for a downstream task, such as classification or regression. This is very useful as it doesn't require any manually labeled data. In this work, we introduce Probabilistic Hyper-Graphs using Masked Autoencoders (PHG-MAE): a novel model that unifies the classical work on neural graphs \cite{leordeanu2021semi} with the modern approach of masked autoencoders under a common theoretical framework. Through random masking of entire modalities, not just patches, the model samples from the distribution of hyper-edges on each forward pass. Additionally, the model adapts the standard MAE algorithm by combining pre-training and fine-tuning into a single training loop. Moreover, our approach enables the creation of inference-time ensembles which, through aggregation, boost the final prediction performance and consistency. Lastly, we show that we can apply knowledge distillation on top of the ensembles with little loss in performance, even with models that have fewer than 1M parameters. While our work mostly focuses on outdoor UAV scenes that contain multiple world interpretations and modalities, the same steps can be followed in other similar domains, such as autonomous driving or indoor robotics. In order to streamline the process of integrating external pre-trained experts for computer vision multi-modal multi-task learning (MTL) scenarios, we developed a data-pipeline software. Using this tool, we have created and released a fully-automated extension of the Dronescapes dataset. All the technical details, code and reproduction steps are publicly released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
