不要以貌取人:迈向运动连贯的视频表示
Don't Judge by the Look: Towards Motion Coherent Video Representation
AI总结:
针对视频理解中色调变化有益于模型关注运动模式的特点,提出数据增强方法MCA及操作SwapMix,并引入VA解决分布偏移以学习外观不变性表示,验证了其有效性与泛化能力。
摘要:
当前目标识别的训练流程在进行数据增强时忽略了Hue Jittering(色调抖动),因为它不仅会带来不利于分类的外观变化,而且其实现效率在实践中也很低。在本研究中,我们在视频理解的背景下研究了色调变化的影响,并发现这种变化是有益的,因为在包含运动信息的视频中,静态外观不那么重要。基于这一观察,我们提出了一种用于视频理解的数据增强方法,命名为Motion Coherent Augmentation (MCA)(运动连贯增强),该方法在视频中引入外观变化,并隐式地鼓励模型优先关注运动模式,而非静态外观。具体而言,我们提出了一种操作SwapMix来高效地修改视频样本的外观,并引入Variation Alignment (VA)(变化对齐)来解决SwapMix引起的分布偏移,强制模型学习外观不变性表示。在各种架构和不同数据集上的全面实证评估,确实验证了MCA的有效性和泛化能力,以及VA在其他增强方法中的应用。代码可在https://github.com/BeSpontaneous/MCA-pytorch获取。
英文摘要:
Current training pipelines in object recognition neglect Hue Jittering when doing data augmentation as it not only brings appearance changes that are detrimental to classification, but also the implementation is inefficient in practice. In this study, we investigate the effect of hue variance in the context of video understanding and find this variance to be beneficial since static appearances are less important in videos that contain motion information. Based on this observation, we propose a data augmentation method for video understanding, named Motion Coherent Augmentation (MCA), that introduces appearance variation in videos and implicitly encourages the model to prioritize motion patterns, rather than static appearances. Concretely, we propose an operation SwapMix to efficiently modify the appearance of video samples, and introduce Variation Alignment (VA) to resolve the distribution shift caused by SwapMix, enforcing the model to learn appearance invariant representations. Comprehensive empirical evaluation across various architectures and different datasets solidly validates the effectiveness and generalization ability of MCA, and the application of VA in other augmentation methods. Code is available at https://github.com/BeSpontaneous/MCA-pytorch.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。