自监督视频预训练产生鲁棒且更符合人类的视觉表征
Self-supervised video pretraining yields robust and more human-aligned visual representations
AI总结：
提出 VITO 自监督视频预训练框架，通过视频整理与对比学习获得跨图像和视频任务、鲁棒且更符合人类判断的统一视觉表征。
摘要：
人类通过观察物体和场景随时间的演变来学习强大的表征。然而，除了需要显式时间理解的特定任务之外，静态图像预训练仍然是学习视觉基础模型的主导范式。我们质疑这种错配，并探讨视频预训练是否能够产生具有人类感知特征的视觉表征：跨任务泛化、对扰动的鲁棒性，以及与人类判断的一致性。为此，我们提出了一种新的视频整理流程，并开发了一个对比框架，从其中的复杂变换中学习。这种从视频中提炼知识的简单范式称为 VITO，可产生通用表征，在图像理解任务上大幅优于以往的视频预训练方法，并在视频理解任务上优于图像预训练方法。此外，与图像、视频以及对抗训练得到的表征相比，VITO 表征对自然和合成形变具有显著更强的鲁棒性。最后，VITO 的预测与人类判断高度一致，超过了专门为此目的训练的模型。这些结果共同表明，视频预训练可能是学习关于视觉世界的统一、鲁棒且符合人类的表征的一种简单方法。
英文摘要：
Humans learn powerful representations of objects and scenes by observing how they evolve over time. Yet, outside of specific tasks that require explicit temporal understanding, static image pretraining remains the dominant paradigm for learning visual foundation models. We question this mismatch, and ask whether video pretraining can yield visual representations that bear the hallmarks of human perception: generalisation across tasks, robustness to perturbations, and consistency with human judgements. To that end we propose a novel procedure for curating videos, and develop a contrastive framework which learns from the complex transformations therein. This simple paradigm for distilling knowledge from videos, called VITO, yields general representations that far outperform prior video pretraining methods on image understanding tasks, and image pretraining methods on video understanding tasks. Moreover, VITO representations are significantly more robust to natural and synthetic deformations than image-, video-, and adversarially-trained ones. Finally, VITO's predictions are strongly aligned with human judgements, surpassing models that were specifically trained for that purpose. Together, these results suggest that video pretraining could be a simple way of learning unified, robust, and human-aligned representations of the visual world.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
