探索更多引导：一种面向手语翻译的任务感知指令网络及其数据增强方法
Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation
AI总结：
本文提出任务感知指令网络TIN-SLT，通过指令模块、特征融合策略和多层次数据增强提升手语翻译性能，并在两个基准数据集上取得领先结果。
摘要：
手语识别与翻译首先使用识别模块从手语视频中生成注释，然后利用翻译模块将注释翻译成口语语句。现有工作大多关注识别步骤，对手语翻译的关注较少。在本工作中，我们提出了一种面向手语翻译的任务感知指令网络，即TIN-SLT，通过将指令模块和基于学习的特征融合策略引入Transformer网络。通过这种方式，预训练模型的语言能力可以被充分探索和利用，从而进一步提升翻译性能。此外，通过探索手语注释和目标口语的表示空间，我们提出了一种多层次数据增强方案来调整训练集的数据分布。我们在两个具有挑战性的基准数据集PHOENIX-2014-T和ASLG-PC12上进行了大量实验，在BLEU-4指标上，我们的方法分别比此前最佳方案高出1.65和1.42。我们的代码已发布在https://github.com/yongcaoplus/TIN-SLT。
英文摘要：
Sign language recognition and translation first uses a recognition module to generate glosses from sign language videos and then employs a translation module to translate glosses into spoken sentences. Most existing works focus on the recognition step, while paying less attention to sign language translation. In this work, we propose a task-aware instruction network, namely TIN-SLT, for sign language translation, by introducing the instruction module and the learning-based feature fuse strategy into a Transformer network. In this way, the pre-trained model's language ability can be well explored and utilized to further boost the translation performance. Moreover, by exploring the representation space of sign language glosses and target spoken language, we propose a multi-level data augmentation scheme to adjust the data distribution of the training set. We conduct extensive experiments on two challenging benchmark datasets, PHOENIX-2014-T and ASLG-PC12, on which our method outperforms former best solutions by 1.65 and 1.42 in terms of BLEU-4. Our code is published at https://github.com/yongcaoplus/TIN-SLT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
