随机胜出:重新思考视觉token的分组策略
Random Wins All: Rethinking Grouping Strategies for Vision Tokens
AI总结:
本文提出随机分组策略,通过简单方法提升视觉token处理效率,实验显示其在多种任务中表现优异。
摘要:
自从Transformer被引入视觉架构以来,其二次复杂度一直是一个重要的问题,许多研究努力旨在解决这个问题。一种代表性的方法是将token分组,在每个组内进行自注意力计算,或在每个组内对token进行池化。为此,已经提出了各种精心设计的分组策略以提高视觉Transformer的性能。然而,我们提出了以下问题:这些精心设计的分组方法真的有必要吗?是否存在一种更简单且更统一的token分组方法可以替代这些多样化的方法?因此,我们提出了随机分组策略,这是一种简单且快速的视觉token分组方法。我们在多个基线上验证了这种方法,并实验表明,随机分组几乎超越了所有其他分组方法。当转移到下游任务,如目标检测时,随机分组显示出更加显著的优势。针对这一现象,我们从多个角度对随机分组的优势进行了详细分析,并识别出几种关键要素:位置信息、头特征多样性、全局感受野和固定分组模式。我们证明只要这四个条件得到满足,视觉token只需要一个极其简单的分组策略就能高效且有效地处理各种视觉任务。我们还验证了所提出的随机方法在多种模态上的有效性,包括视觉任务、点云处理和视觉-语言模型。代码将在https://github.com/qhfan/random上提供。
英文摘要:
Since Transformers are introduced into vision architectures, their quadratic complexity has always been a significant issue that many research efforts aim to address. A representative approach involves grouping tokens, performing self-attention calculations within each group, or pooling the tokens within each group into a single token. To this end, various carefully designed grouping strategies have been proposed to enhance the performance of Vision Transformers. Here, we pose the following questions: \textbf{Are these carefully designed grouping methods truly necessary? Is there a simpler and more unified token grouping method that can replace these diverse methods?} Therefore, we propose the random grouping strategy, which involves a simple and fast random grouping strategy for vision tokens. We validate this approach on multiple baselines, and experiments show that random grouping almost outperforms all other grouping methods. When transferred to downstream tasks, such as object detection, random grouping demonstrates even more pronounced advantages. In response to this phenomenon, we conduct a detailed analysis of the advantages of random grouping from multiple perspectives and identify several crucial elements for the design of grouping strategies: positional information, head feature diversity, global receptive field, and fixed grouping pattern. We demonstrate that as long as these four conditions are met, vision tokens require only an extremely simple grouping strategy to efficiently and effectively handle various visual tasks. We also validate the effectiveness of our proposed random method across multiple modalities, including visual tasks, point cloud processing, and vision-language models. Code will be available at https://github.com/qhfan/random.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。