优化音素唇读系统中的视觉单位选择
Alternative Visual Units for an Optimized Phoneme-Based Lipreading System
摘要：
唇读是通过观察嘴唇的运动来理解言语。观察到的一系列嘴唇运动是视觉唇姿态的有序序列。这些姿态是众所周知的，但尚未正式定义为“视位”。在这篇文章中，我们描述了一种结构化的方法，它允许我们创建一个固定数量的视位在每个集合内的说话者相关的视位。我们创建大小为2到45的视位集。每个视位集合都基于音素聚类，因此每个集合具有唯一的音素到视位映射。我们首先提出了一个实验，使用这些地图和资源管理视听（RMAV）数据集，显示了改变视位图的大小在依赖于扬声器的机器唇读的效果，并证明了音素分类器的单词识别是可能的。此外，我们还发现视位和音素之间存在着更好的中间单位。其次，我们提出了一种新的两遍训练方案的音素分类器。这种方法使用我们在第一次实验中的新的中间视觉单位作为分类器;在使用音素到视位映射之前，我们将这些重新训练成音素分类器。该方法显着改善了以前的唇读结果与RMAV扬声器。
英文摘要：
Lipreading is understanding speech from observed lip movements. An observed series of lip motions is an ordered sequence of visual lip gestures. These gestures are commonly known, but as yet are not formally defined, as `visemes'. In this article, we describe a structured approach which allows us to create speaker-dependent visemes with a fixed number of visemes within each set. We create sets of visemes for sizes two to 45. Each set of visemes is based upon clustering phonemes, thus each set has a unique phoneme-to-viseme mapping. We first present an experiment using these maps and the Resource Management Audio-Visual (RMAV) dataset which shows the effect of changing the viseme map size in speaker-dependent machine lipreading and demonstrate that word recognition with phoneme classifiers is possible. Furthermore, we show that there are intermediate units between visemes and phonemes which are better still. Second, we present a novel two-pass training scheme for phoneme classifiers. This approach uses our new intermediary visual units from our first experiment in the first pass as classifiers; before using the phoneme-to-viseme maps, we retrain these into phoneme classifiers. This method significantly improves on previous lipreading results with RMAV speakers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
