Sign language to text is a crucial technology that can break down communication barriers for individuals with hearing difficulties. We replicate and try to improve on a recently published study. We evaluate models using BLEU and rBLEU metrics to ensure translation quality. During our ablation study, we found that the model's performance is significantly influenced by optimizers, activation functions, and label smoothing. Further research aims to refine visual feature capturing, enhance decoder utilization, and integrate pre-trained decoders for better translation outcomes. Our source code is available to facilitate replication of our results and encourage future research.
We outline a bidirectional translation system that converts sentences from American Sign Language (ASL) to English, and vice versa. To perform machine translation between ASL and English, we utilize a generative approach. Specifically, we employ an adjustment to the IBM word-alignment model 1 (IBM WAM1), where we define language models for English and ASL, as well as a translation model, and attempt to generate a translation that maximizes the posterior distribution defined by these models. Then, using these models, we are able to quantify the concepts of fluency and faithfulness of a translat…
Research Lab. UTIC, University of Tunis（突尼斯大学UTIC研究实验室）
This works aims to design a statistical machine translation from English text to American Sign Language (ASL). The system is based on Moses tool with some modifications and the results are synthesized through a 3D avatar for interpretation. First, we translate the input text to gloss, a written form of ASL. Second, we pass the output to the WebSign Plug-in to play the sign. Contributions of this work are the use of a new couple of language English/ASL and an improvement of statistical machine translation based on string matching thanks to Jaro-distance.
TTI-Chicago（芝加哥丰田技术研究所）；Univeristy of Chicago（芝加哥大学）
Natural language processing for sign language video - including tasks like recognition, translation, and search - is crucial for making artificial intelligence technologies accessible to deaf individuals, and is gaining research interest in recent years. In this paper, we address the problem of searching for fingerspelled key-words or key phrases in raw sign language videos. This is an important task since significant content in sign language is often conveyed via fingerspelling, and to our knowledge the task has not been studied before. We propose an end-to-end model for this task, FSS-Net, t…
University of Zurich（苏黎世大学）；Bar-Ilan University（巴伊兰大学）
This paper presents work on novel machine translation (MT) systems between spoken and signed languages, where signed languages are represented in SignWriting, a sign language writing system. Our work seeks to address the lack of out-of-the-box support for signed languages in current MT systems and is based on the SignBank dataset, which contains pairs of spoken language text and SignWriting content. We introduce novel methods to parse, factorize, decode, and evaluate SignWriting, leveraging ideas from neural factored MT. In a bilingual setup--translating from American Sign Language to (America…
Kreyòl-MT: Building MT for Latin American, Caribbean and Colonial African Creole Languages
Johns Hopkins University（约翰斯·霍普金斯大学）；National Institute of Information and Communications Technology（信息通信研究机构）；Brigham Young University（杨百翰大学）；Inria Paris（法国国家信息与自动化研究所巴黎分部）；Nile University of Nigeria（尼日利亚尼罗河大学）；Université Paris Nanterre（巴黎第十大学）；University of Minnesota - Twin Cities（明尼苏达大学双城分校）；University of Toronto（多伦多大学）
Existing work on sign language translation - that is, translation from sign language videos into sentences in a written language - has focused mainly on (1) data collected in a controlled environment or (2) data in a specific domain, which limits the applicability to real-world settings. In this paper, we introduce OpenASL, a large-scale American Sign Language (ASL) - English dataset collected from online video sites (e.g., YouTube). OpenASL contains 288 hours of ASL videos in multiple domains from over 200 signers and is the largest publicly available ASL translation dataset to date. To tackl…
Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）
In this thesis, we study the problem of recognizing video sequences of fingerspelled letters in American Sign Language (ASL). Fingerspelling comprises a significant but relatively understudied part of ASL, and recognizing it is challenging for a number of reasons: It involves quick, small motions that are often highly coarticulated; it exhibits significant variation between signers; and there has been a dearth of continuous fingerspelling data collected. In this work, we propose several types of recognition approaches, and explore the signer variation problem. Our best-performing models are se…
According to interviews with people who work with speech impaired persons, speech impaired people have difficulties in communicating with other people around them who do not know the sign language, and this situation may cause them to isolate themselves from society and lose their sense of independence. With this paper, to increase the quality of life of individuals with facilitating communication between individuals who use sign language and who do not know this language, a new American Sign Language (ASL) digits dataset that can help to create machine learning algorithms which need to large …
Toyota Technological Institute at Chicago（芝加哥丰田技术学院）；University of Chicago（芝加哥大学）
Fingerspelling, in which words are signed letter by letter, is an important component of American Sign Language. Most previous work on automatic fingerspelling recognition has assumed that the boundaries of fingerspelling regions in signing videos are known beforehand. In this paper, we consider the task of fingerspelling detection in raw, untrimmed sign language videos. This is an important step towards building real-world fingerspelling recognition systems. We propose a benchmark and a suite of evaluation metrics, some of which reflect the effect of detection on the downstream fingerspelling…
Sign language is an essential resource enabling access to communication and proper socioemotional development for individuals suffering from disabling hearing loss. As this population is expected to reach 700 million by 2050, the importance of the language becomes even more essential as it plays a critical role to ensure the inclusion of such individuals in society. The Sign Language Recognition field aims to bridge the gap between users and non-users of sign languages. However, the scarcity in quantity and quality of datasets is one of the main challenges limiting the exploration of novel app…
Toyota Technological Institute at Chicago（芝加哥丰田技术研究院）；University of Chicago（芝加哥大学）
We address the problem of American Sign Language fingerspelling recognition in the wild, using videos collected from websites. We introduce the largest data set available so far for the problem of fingerspelling recognition, and the first using naturally occurring video data. Using this data set, we present the first attempt to recognize fingerspelling sequences in this challenging setting. Unlike prior work, our video data is extremely challenging due to low frame rates and visual variability. To tackle the visual challenges, we train a special-purpose signing hand detector using a small subs…
Bar-Ilan University（巴伊兰大学）；Carnegie Mellon University（卡内基梅隆大学）；Allen Institute for AI（艾伦人工智能研究所）
Sign language translation (SLT) is often decomposed into video-to-gloss recognition and gloss-to-text translation, where a gloss is a sequence of transcribed spoken-language words in the order in which they are signed. We focus here on gloss-to-text translation, which we treat as a low-resource neural machine translation (NMT) problem. However, unlike traditional low-resource NMT, gloss-to-text translation differs because gloss-text pairs often have a higher lexical overlap and lower syntactic overlap than pairs of spoken languages. We exploit this lexical overlap and handle syntactic divergen…
First Attempt at Building Parallel Corpora for Machine Translation of Northeast India's Very Low-Resource Languages
University of Colorado, Colorado Springs（科罗拉多大学科罗拉多斯普林斯分校）；Instituto Politécnico Nacional（墨西哥国立理工学院）；Lelapa AI（莱拉帕人工智能公司）；Palmer Ridge High School（帕默里奇高中）
Machine translation (MT) involving Indigenous languages, including those possibly endangered, is challenging due to lack of sufficient parallel data. We describe an approach exploiting bilingual and multilingual pretrained MT models in a transfer learning setting to translate from Spanish to ten South American Indigenous languages. Our models set new SOTA on five out of the ten language pairs we consider, even doubling performance on one of these five pairs. Unlike previous SOTA that perform data augmentation to enlarge the train sets, we retain the low-resource setting to test the effectivene…
Boston University（波士顿大学）；Rutgers University（罗格斯大学）
The American Sign Language Linguistic Research Project (ASLLRP) provides Internet access to high-quality ASL video data, generally including front and side views and a close-up of the face. The manual and non-manual components of the signing have been linguistically annotated using SignStream(R). The recently expanded video corpora can be browsed and searched through the Data Access Interface (DAI 2) we have designed; it is possible to carry out complex searches. The data from our corpora can also be downloaded; annotations are available in an XML export format. We have also developed the ASLL…
Rochester Institute of Technology（罗切斯特理工学院）；The City College of New York（纽约城市学院）；Hunter College, City University of New York（纽约市立大学亨特学院）
We are releasing a dataset containing videos of both fluent and non-fluent signers using American Sign Language (ASL), which were collected using a Kinect v2 sensor. This dataset was collected as a part of a project to develop and evaluate computer vision algorithms to support new technologies for automatic detection of ASL fluency attributes. A total of 45 fluent and non-fluent participants were asked to perform signing homework assignments that are similar to the assignments used in introductory or intermediate level ASL courses. The data is annotated to identify several aspects of signing i…
SRM Institute of Science and Technology（SRM科学与技术学院）；Cognitive Applications Research Lab（认知应用研究实验室）
Due to the boom in technical compute in the last few years, the world has seen massive advances in artificially intelligent systems solving diverse real-world problems. But a major roadblock in the ubiquitous acceptance of these models is their enormous computational complexity and memory footprint. Hence efficient architectures and training techniques are required for deployment on extremely low resource inference endpoints. This paper proposes an architecture for detection of alphabets in American Sign Language on an ARM Cortex-M7 microcontroller having just 496 KB of framebuffer RAM. Levera…
Targeted Linguistic Analysis of Sign Language Models with Minimal Translation Pairs
Toyota Technological Institute at Chicago（芝加哥丰田技术研究所）；Linguistics Department, The University of Texas at Austin（德克萨斯大学奥斯汀分校语言学系）；Linguistics Department, The University of Chicago（芝加哥大学语言学系）
Cherokee is a highly endangered Native American language spoken by the Cherokee people. The Cherokee culture is deeply embedded in its language. However, there are approximately only 2,000 fluent first language Cherokee speakers remaining in the world, and the number is declining every year. To help save this endangered language, we introduce ChrEn, a Cherokee-English parallel dataset, to facilitate machine translation research between Cherokee and English. Compared to some popular machine translation language pairs, ChrEn is extremely low-resource, only containing 14k sentence pairs in total.…
Despite tremendous progress in natural language processing using deep learning techniques in recent years, sign language production and comprehension has advanced very little. One critical barrier is the lack of largescale datasets available to the public due to the unbearable cost of labeled data generation. Efforts to provide public data for American Sign Language (ASL) comprehension have yielded two datasets, comprising more than thousand video clips. These datasets are large enough to enable a meaningful start to deep learning research on sign languages but are far too small to lead to any…
ADAT: Time-Series-Aware Adaptive Transformer Architecture for Sign Language Translation
College of Information Technology, United Arab Emirates University（阿联酋大学信息技术学院）；Emirates Center for Mobility Research, United Arab Emirates University（阿联酋大学阿联酋移动性研究中心）
The Ongiini-Eval-OW Benchmark: A Concept Paper for the Planned Benchmarking of Machine Translation and Large Language Models on Oshindonga and Oshikwanyama
From Binary to Bilingual: How the National Weather Service is Using Artificial Intelligence to Develop a Comprehensive Translation Program
Department of Climate, Meteorology & Atmospheric Sciences, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校气候、气象与大气科学系）；Department of Communication, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校通信系）；NOAA/National Weather Service Office of Central Processing（国家海洋和大气管理局/国家气象服务中央处理办公室）；Department of Geography & Geographic Information Science, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校地理与地理信息科学系）；Department of Atmospheric Sciences, University of North Dakota（北达科塔大学大气科学系）；Pace University（帕克大学）；NOAA/National Weather Service Office of Observations（国家海洋和大气管理局/国家气象服务观测办公室）；LILT, Emeryville, CA（LILT，埃默里维尔，CA）；NOAA/National Weather Service（国家海洋和大气管理局/国家气象服务）；NOAA/NWS Southeast River Forecast Center（国家海洋和大气管理局/国家气象服务东南河预报中心）；Cooperative Institute for Research in the Atmosphere, Colorado State University（大气研究合作研究所，科罗拉多州立大学）；Center for Spatial Analysis, University of Oklahoma（空间分析中心，俄克拉荷马大学）
