BEST：BERT预训练用于耦合符号化的手语识别
BEST: BERT Pre-Training for Sign Language Recognition with Coupling Tokenization
摘要：
在这项工作中，我们致力于利用BERT预训练的成功和建模的特定领域的统计，以丰富的手语识别~（SLR）模型。考虑到手和身体在手语表达中的主导地位，我们将它们组织为姿势三元组单元，并以帧方式将它们馈送到Transformer骨干中。预训练是通过从损坏的输入序列中重建被掩蔽的三元组单元来执行的，其学习内部和外部三元组单元之间的分层相关上下文线索。值得注意的是，不同于BERT中的高语义单词标记，姿态单元是最初位于连续空间中的低电平信号，这阻止了直接采用BERT交叉熵目标。为此，我们通过耦合标记化的三联体单元来弥合这一语义鸿沟。它自适应地从姿势三元组单元中提取表示语义手势/身体状态的离散伪标签。在预训练之后，我们在下游SLR任务上微调预训练的编码器，以及新添加的特定于任务的层。进行了大量的实验，以验证我们提出的方法的有效性，实现了新的国家的最先进的性能，所有四个基准具有显着的增益。
英文摘要：
In this work, we are dedicated to leveraging the BERT pre-training success and modeling the domain-specific statistics to fertilize the sign language recognition~(SLR) model. Considering the dominance of hand and body in sign language expression, we organize them as pose triplet units and feed them into the Transformer backbone in a frame-wise manner. Pre-training is performed via reconstructing the masked triplet unit from the corrupted input sequence, which learns the hierarchical correlation context cues among internal and external triplet units. Notably, different from the highly semantic word token in BERT, the pose unit is a low-level signal originally located in continuous space, which prevents the direct adoption of the BERT cross-entropy objective. To this end, we bridge this semantic gap via coupling tokenization of the triplet unit. It adaptively extracts the discrete pseudo label from the pose triplet unit, which represents the semantic gesture/body state. After pre-training, we fine-tune the pre-trained encoder on the downstream SLR task, jointly with the newly added task-specific layer. Extensive experiments are conducted to validate the effectiveness of our proposed method, achieving new state-of-the-art performance on all four benchmarks with a notable gain.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
