基于3D虚拟形象的语音到手语翻译的简单基线
A Simple Baseline for Spoken Language to Sign Language Translation with 3D Avatars
AI总结：
本文首次提出以3D手语为输出格式的Spoken2Sign翻译任务，通过构建词汇-3D手语词典并训练Text2Gloss翻译器、手语连接器和渲染模块，实现口语到3D手语虚拟形象的翻译。
摘要：
本文的目标是开发一个将口语翻译成手语的功能性系统，称为Spoken2Sign翻译。Spoken2Sign任务与传统的手语到口语（Sign2Spoken）翻译是正交且互补的。为了实现Spoken2Sign翻译，我们提出了一个由三个步骤组成的简单基线：1）利用现有的Sign2Spoken基准构建词汇-视频词典；2）为词典中的每个手语视频估计一个3D手语动作；3）借助生成的词汇-3D手语词典，训练一个Spoken2Sign模型，该模型由Text2Gloss翻译器、手语连接器和渲染模块组成。翻译结果随后通过手语虚拟形象进行展示。据我们所知，我们是首个以3D手语作为输出格式提出Spoken2Sign任务的工作。除了具备Spoken2Sign翻译能力外，我们还证明了该方法的两个副产品——3D关键点增强和多视角理解——能够辅助基于关键点的手语理解。代码和模型可在https://github.com/FangyunWei/SLRT获取。
英文摘要：
The objective of this paper is to develop a functional system for translating spoken languages into sign languages, referred to as Spoken2Sign translation. The Spoken2Sign task is orthogonal and complementary to traditional sign language to spoken language (Sign2Spoken) translation. To enable Spoken2Sign translation, we present a simple baseline consisting of three steps: 1) creating a gloss-video dictionary using existing Sign2Spoken benchmarks; 2) estimating a 3D sign for each sign video in the dictionary; 3) training a Spoken2Sign model, which is composed of a Text2Gloss translator, a sign connector, and a rendering module, with the aid of the yielded gloss-3D sign dictionary. The translation results are then displayed through a sign avatar. As far as we know, we are the first to present the Spoken2Sign task in an output format of 3D signs. In addition to its capability of Spoken2Sign translation, we also demonstrate that two by-products of our approach-3D keypoint augmentation and multi-view understanding-can assist in keypoint-based sign language understanding. Code and models are available at https://github.com/FangyunWei/SLRT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
