arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.00780cs.CV
收藏

SpiralFovea: 输入自适应的中央凹令牌化作为资源自适应推理的第三杠杆

SpiralFovea: Input-Adaptive Foveated Tokenization as a Third Lever of Resource-Adaptive Inference

Kyan Mahajan, Mohammad Saqlain

AI总结:

SpiralFovea提出输入自适应令牌化方法,根据局部视觉熵动态选择令牌的身份、位置、尺度和数量,在细粒度分类任务中以60%更少令牌提升准确率1.7-2.1个百分点,并降低84%自注意力计算量。

摘要:

大多数基础模型的自适应推理技术改变模型的行为——提前退出、MoE路由、KV缓存压缩、动态注意力稀疏性。然而,输入到主干网络的数据仍然是固定网格的令牌化,与图像内容无关。我们认为这是一个被忽视的杠杆。我们提出SpiralFovea,一种无参数、输入自适应的令牌化器,其中令牌的身份、位置、尺度和数量都是局部视觉熵的函数,并且选择在任何主干网络参数被查询之前完成。围绕内容驱动的热点锚点,多尺度螺旋环产生≤78个补丁,在输入阶段替换标准的196补丁ViT网格。在四个经典的细粒度基准上,SpiralFovea相比匹配的静态令牌化基线,准确率提升1.7-2.1个百分点,输入令牌减少60%,每个Transformer层的自注意力FLOPs降低84%,吞吐量提升18-29%。在CUB-200-2011 Genus上跨四个主干网络的控制消融实验揭示了一个清晰的诊断:增益幅度与主干网络的全图像位置先验强度成反比,将自监督基础模型定位为输入自适应令牌化最有价值的领域。

英文摘要:

Most adaptive-inference techniques for foundation models change what the model does - early exit, MoE routing, KV-cache compression, dynamic attention sparsity. The input that hits the backbone, however, remains a fixed-grid tokenisation indifferent to image content. We argue that this is a missed lever. We present SpiralFovea, a parameter-free, input-adaptive tokeniser in which token identity, location, scale, and count are all functions of local visual entropy and selection completes before any backbone parameter is queried. Around content-driven hotspot anchors, multi-scale spiral rings produce <= 78 patches that replace the standard 196-patch ViT grid at the input stage. Across four canonical fine-grained benchmarks, SpiralFovea yields +1.7-2.1 pp accuracy with a 60% reduction in input tokens, an 84% reduction in self-attention FLOPs at every transformer layer, and 18-29% throughput gains over the matched static tokenisation baseline. A controlled ablation on CUB-200-2011 Genus across four backbones reveals a clean diagnostic: the gain magnitude tracks inversely with the strength of the backbone's whole-image positional prior, isolating self-supervised foundation models as the regime where input-adaptive tokenisation is most valuable.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑