表面网格是表示人类皮层结构和功能信息的常用域,但其复杂的拓扑结构和几何形态给深度学习分析带来了重大挑战。尽管 Transformer 已作为适用于序列到序列学习的领域无关架构表现出色,但自注意力操作的二次成本仍是许多密集预测任务的障碍。受视觉 Transformer 分层建模最新进展的启发,我们提出多尺度表面 Vision Transformer(Multiscale Surface Vision Transformer,MS-SiT),作为表面深度学习的骨干架构。自注意力机制在局部网格窗口内应用,以实现对底层数据的高分辨率采样,同时移位窗口策略改善了窗口之间的信息共享。相邻 patch 被相继合并,使 MS-SiT 能够学习适用于任何预测任务的分层表示。结果表明,在使用 Developing Human Connectome Project(dHCP)数据集进行新生儿表型预测任务时,MS-SiT 优于现有表面深度学习方法。此外,将 MS-SiT 骨干构建为用于表面分割的 U 形架构后,在使用 UK Biobank(UKB)和人工标注 MindBoggle 数据集进行皮层分区时取得了具有竞争力的结果。代码和训练模型公开发布于 https://github.com/metrics-lab/surface-vision-transformers。
英文摘要:
Surface meshes are a favoured domain for representing structural and functional information on the human cortex, but their complex topology and geometry pose significant challenges for deep learning analysis. While Transformers have excelled as domain-agnostic architectures for sequence-to-sequence learning, the quadratic cost of the self-attention operation remains an obstacle for many dense prediction tasks. Inspired by some of the latest advances in hierarchical modelling with vision transformers, we introduce the Multiscale Surface Vision Transformer (MS-SiT) as a backbone architecture for surface deep learning. The self-attention mechanism is applied within local-mesh-windows to allow for high-resolution sampling of the underlying data, while a shifted-window strategy improves the sharing of information between windows. Neighbouring patches are successively merged, allowing the MS-SiT to learn hierarchical representations suitable for any prediction task. Results demonstrate that the MS-SiT outperforms existing surface deep learning methods for neonatal phenotyping prediction tasks using the Developing Human Connectome Project (dHCP) dataset. Furthermore, building the MS-SiT backbone into a U-shaped architecture for surface segmentation demonstrates competitive results on cortical parcellation using the UK Biobank (UKB) and manually-annotated MindBoggle datasets. Code and trained models are publicly available at https://github.com/metrics-lab/surface-vision-transformers.
Explicitly Increasing Input Information Density for Vision Transformers on Small Datasets
University of Kansas(堪萨斯大学);Beijing Jiaotong University(北京交通大学);OPPO US Research Center, InnoPeak Technology Inc(英诺皮克科技有限公司OPPO美国研究中心);Toronto Metropolitan University(多伦多都会大学)
AutoTaskFormer: Searching Vision Transformers for Multi-task Learning
Zhejiang University(浙江大学);Michigan State University(密歇根州立大学);Microsoft Research Asia(微软亚洲研究院);Vrije Universiteit Amsterdam(阿姆斯特丹自由大学);The Ohio State University(俄亥俄州立大学)
MSV-Mamba: A Multiscale Vision Mamba Network for Echocardiography Segmentation
School of Computer and Information Engineering, Shanghai Polytechnic University(上海第二工业大学计算机与信息工程学院);The First Affiliated Hospital of Jinan University(暨南大学附属第一医院);Fudan University Huashan Hospital(复旦大学附属华山医院)