状态空间模型是有效的手语学习者:利用语音学的组合性进行词汇级识别
State Space Models are Effective Sign Language Learners: Exploiting Phonological Compositionality for Vocabulary-Scale Recognition
AI总结:
本文提出PHONSSM模型,通过语音学分解提升手语识别性能,实现词汇级识别,优于现有方法,尤其在少样本场景下表现更佳。
摘要:
手语识别面临灾难性扩展失败:在小词汇量上表现优异的模型在现实规模下崩溃。现有架构将手语视为原子视觉模式,学习扁平表示无法利用手语的组合结构——系统性组织的离散语音学参数(手形、位置、运动、方向)在词汇中被重复使用。我们引入PHONSSM,通过解剖学基础的图注意力机制、显式分解到正交子空间以及原型分类实现语音学分解,使在最大ASL数据集(5565个手语)上仅使用骨架数据时,在WLASL2000上达到72.1%(比骨架SOTA高18.4个百分点),超越大多数RGB方法而无需视频输入。在少样本场景中增益最为显著(+225%相对),且模型能零样本迁移到ASL Citizen,超越监督RGB基线。词汇扩展瓶颈本质上是表征学习问题,可通过模仿语言结构的组合归纳偏置解决。
英文摘要:
Sign language recognition suffers from catastrophic scaling failure: models achieving high accuracy on small vocabularies collapse at realistic sizes. Existing architectures treat signs as atomic visual patterns, learning flat representations that cannot exploit the compositional structure of sign languages-systematically organized from discrete phonological parameters (handshape, location, movement, orientation) reused across the vocabulary. We introduce PHONSSM, enforcing phonological decomposition through anatomically-grounded graph attention, explicit factorization into orthogonal subspaces, and prototypical classification enabling few-shot transfer. Using skeleton data alone on the largest ASL dataset ever assembled (5,565 signs), PHONSSM achieves 72.1% on WLASL2000 (+18.4pp over skeleton SOTA), surpassing most RGB methods without video input. Gains are most dramatic in the few-shot regime (+225% relative), and the model transfers zero-shot to ASL Citizen, exceeding supervised RGB baselines. The vocabulary scaling bottleneck is fundamentally a representation learning problem, solvable through compositional inductive biases mirroring linguistic structure.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。