arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2111.12993cs.CVcs.LG
收藏

PolyViT:图像、视频和音频方面的共同培训视觉Transformer

PolyViT: Co-training Vision Transformers on Images, Videos and Audio

University of Cambridge(剑桥大学);Google Research(谷歌研究院);Alan Turing Institute(阿兰·图灵研究所)

Valerii Likhosherstov, Anurag Arnab, Krzysztof Choromanski, Mario Lucic, Yi Tay, Adrian Weller, Mostafa Dehghani

摘要:

我们能否训练一个能够处理多个模态和数据集的单个Transformer模型,同时共享几乎所有可学习的参数?我们提出了PolyViT,一个在图像,音频和视频上训练的模型,它回答了这个问题。通过在单个模态上共同训练不同的任务,我们能够提高每个任务的准确性,并在5个标准视频和音频分类数据集上实现最先进的结果。在多个模态和任务上共同训练PolyViT,可以产生一个参数效率更高的模型,并学习跨多个领域的泛化表示。此外,我们还证明了协同训练简单实用,因为我们不需要为每个数据集组合调整超参数,而是可以简单地调整标准的单任务训练。

英文摘要:

Can we train a single transformer model capable of processing multiple modalities and datasets, whilst sharing almost all of its learnable parameters? We present PolyViT, a model trained on image, audio and video which answers this question. By co-training different tasks on a single modality, we are able to improve the accuracy of each individual task and achieve state-of-the-art results on 5 standard video- and audio-classification datasets. Co-training PolyViT on multiple modalities and tasks leads to a model that is even more parameter-efficient, and learns representations that generalize across multiple domains. Moreover, we show that co-training is simple and practical to implement, as we do not need to tune hyperparameters for each combination of datasets, but can simply adapt those from standard, single-task training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑