唇读中同音异义词的困惑分析
Disentangling Homophemes in Lip Reading using Perplexity Analysis
摘要：
与使用ASCII字符和单词相比，使用视位作为分类模式的自动唇读的性能取得了较小的成功，这主要是由于不同单词共享相同视位的问题。生成式预训练Transformer是一种有效的自回归语言模型，用于自然语言处理中的许多任务，包括句子预测和文本分类。本文提出了一个新的应用，并将其应用于唇读的背景下，它作为一个语言模型转换的视觉语音形式的视位，语言的形式的单词和句子。该网络使用搜索最佳困惑来执行视位到单词的映射，因此是一个解决一对多映射问题的方案，该问题存在于各种单词中，这些单词在说话时听起来不同，看起来相同。本文提出了一种方法来解决一对多的映射问题时，执行自动唇读仅使用视觉线索在两个单独的情况下：第一种情况是在单词的边界，即开始和结束一个词，是未知的;和第二种情况是在边界是已知的。来自基准BBC数据集"Lip Reading Sentences in the Wild"（LRS 2）的句子被分类为具有10.7%的字符出错率和18.0%的单词出错率。本文的主要贡献是提出了一种方法，预测的话，通过使用困惑分析时，只有视觉线索，使用自回归语言模型。
英文摘要：
The performance of automated lip reading using visemes as a classification schema has achieved less success compared with the use of ASCII characters and words largely due to the problem of different words sharing identical visemes. The Generative Pre-Training transformer is an effective autoregressive language model used for many tasks in Natural Language Processing, including sentence prediction and text classification. This paper proposes a new application for this model and applies it in the context of lip reading, where it serves as a language model to convert visual speech in the form of visemes, to language in the form of words and sentences. The network uses the search for optimal perplexity to perform the viseme-to-word mapping and is thus a solution to the one-to-many mapping problem that exists whereby various words that sound different when spoken look identical. This paper proposes a method to tackle the one-to-many mapping problem when performing automated lip reading using solely visual cues in two separate scenarios: the first scenario is where the word boundary, that is, the beginning and the ending of a word, is unknown; and the second scenario is where the boundary is known. Sentences from the benchmark BBC dataset "Lip Reading Sentences in the Wild"(LRS2), are classified with a character error rate of 10.7% and a word error rate of 18.0%. The main contribution of this paper is to propose a method of predicting words through the use of perplexity analysis when only visual cues are present, using an autoregressive language model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
