arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2208.09796cs.CVcs.CY
收藏

走向唇读MOOC:使用合成说话头训练人类唇读

Towards MOOCs for Lipreading: Using Synthetic Talking Heads to Train Humans in Lipreading at Scale

IIIT Hyderabad(海得拉巴国际信息技术研究所);University of Bath(巴斯大学)

Aditya Agarwal, Bipasha Sen, Rudrabha Mukhopadhyay, Vinay Namboodiri, C. V Jawahar

摘要:

许多患有某种形式听力损失的人认为唇读是他们日常交流的主要方式。然而,寻找资源来学习或提高一个人的唇读技能可能是具有挑战性的。在COVID-19大流行期间,由于限制与同龄人和语言治疗师的直接互动,这一情况进一步恶化。如今,在线MOOC平台,如Coursera和Udemy,已成为许多类型技能发展的最有效培训形式。然而,在线唇读资源是稀缺的,因为创建这样的资源是一个广泛的过程,需要数月的人工努力来记录雇用的演员。由于手动管道,这些平台在词汇、支持的语言、口音和扬声器方面也受到限制,并且具有很高的使用成本。在这项工作中,我们研究的可能性,取代真正的人类说话的视频与合成生成的视频。合成数据可以很容易地包含更大的词汇表、口音的变化,甚至本地语言和许多说话者。我们提出了一个端到端的自动化流水线开发这样一个平台,使用国家的最先进的说话头视频生成器网络,文本到语音模型,和计算机视觉技术。然后,我们进行了广泛的人类评估,使用经过深思熟虑的唇读练习,以验证我们设计的平台对现有唇读平台的质量。我们的研究具体指出了我们的方法在开发大规模唇读MOOC平台方面的潜力,该平台可以影响数百万听力损失患者。

英文摘要:

Many people with some form of hearing loss consider lipreading as their primary mode of day-to-day communication. However, finding resources to learn or improve one's lipreading skills can be challenging. This is further exacerbated in the COVID19 pandemic due to restrictions on direct interactions with peers and speech therapists. Today, online MOOCs platforms like Coursera and Udemy have become the most effective form of training for many types of skill development. However, online lipreading resources are scarce as creating such resources is an extensive process needing months of manual effort to record hired actors. Because of the manual pipeline, such platforms are also limited in vocabulary, supported languages, accents, and speakers and have a high usage cost. In this work, we investigate the possibility of replacing real human talking videos with synthetically generated videos. Synthetic data can easily incorporate larger vocabularies, variations in accent, and even local languages and many speakers. We propose an end-to-end automated pipeline to develop such a platform using state-of-the-art talking head video generator networks, text-to-speech models, and computer vision techniques. We then perform an extensive human evaluation using carefully thought out lipreading exercises to validate the quality of our designed platform against the existing lipreading platforms. Our studies concretely point toward the potential of our approach in developing a large-scale lipreading MOOC platform that can impact millions of people with hearing loss.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑